{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced MLP\n",
    "- Advanced techniques for training neural networks\n",
    "    - Weight Initialization\n",
    "    - Nonlinearity (Activation function)\n",
    "    - Optimizers\n",
    "    - Batch Normalization\n",
    "    - Dropout (Regularization)\n",
    "    - Model Ensemble"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.utils.np_utils import to_categorical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Dataset\n",
    "- MNIST dataset\n",
    "- source: http://yann.lecun.com/exdb/mnist/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "(X_train, y_train), (X_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADn9JREFUeJzt3X9sXfV5x/HPU8dxlhDauCmeSzMSIC3QsIbtKoCIgImR\npQgpoKqhUVWljDVdC3RsmQTLpjWb2JRNLVXKGJJZsyQVv0oLIn+wVmBV0GrgYbIQfpVfwV0TjE1w\nIYHSxLGf/eGTygXf73XuPfeeaz/vl2T53vOcc8+jk3x87r3fe8/X3F0A4vlA0Q0AKAbhB4Ii/EBQ\nhB8IivADQRF+ICjCDwRF+IGgCD8Q1IxG7mymtfkszWnkLoFQfq13dNgP2WTWrSn8ZrZS0mZJLZL+\nw903pdafpTk62y6qZZcAEnq8e9LrVv2038xaJN0i6dOSzpC0xszOqPbxADRWLa/5l0l6yd33uPth\nSXdJWpVPWwDqrZbwnyjpF+Pu782W/RYzW2dmvWbWO6xDNewOQJ7q/m6/u3e5e8ndS61qq/fuAExS\nLeHfJ2nBuPsfy5YBmAJqCf/jkhab2SIzmynpc5J25NMWgHqreqjP3Y+Y2TWSfqSxob4t7v5Mbp0B\nqKuaxvnd/QFJD+TUC4AG4uO9QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ER\nfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANB\nEX4gKMIPBFXTLL1m1ifpoKQRSUfcvZRHU8iPzUj/E7d8ZH5d9//8Xy8sWxuZPZrc9qRTBpP12V+1\nZP21m2aWre0s3Z3cdv/IO8n62fesT9ZP/avHkvVmUFP4M3/k7vtzeBwADcTTfiCoWsPvkh4ysyfM\nbF0eDQFojFqf9i93931mdoKkB83sZ+7+yPgVsj8K6yRplmbXuDsAeanpzO/u+7Lfg5Luk7RsgnW6\n3L3k7qVWtdWyOwA5qjr8ZjbHzOYevS1phaSn82oMQH3V8rS/Q9J9Znb0ce5w9x/m0hWAuqs6/O6+\nR9Kncuxl2mo5fXGy7m2tyfqrF3woWX/3nPJj0u0fTI9X/+RT6fHuIv3Xr+Ym6//ybyuT9Z4z7yhb\ne2X43eS2mwYuTtY/+hNP1qcChvqAoAg/EBThB4Ii/EBQhB8IivADQeXxrb7wRi78g2T9pq23JOsf\nby3/1dPpbNhHkvW/v/mLyfqMd9LDbefec03Z2tx9R5Lbtu1PDwXO7u1J1qcCzvxAUIQfCIrwA0ER\nfiAowg8ERfiBoAg/EBTj/Dloe/7VZP2JXy9I1j/eOpBnO7la339Osr7n7fSlv7ee8v2ytbdG0+P0\nHd/+72S9nqb+F3Yr48wPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0GZe+NGNI+3dj/bLmrY/prF0JXn\nJusHVqYvr92y+7hk/cmv3nzMPR114/7fT9YfvyA9jj/y5lvJup9b/urufV9LbqpFa55Mr4D36fFu\nHfCh9NzlGc78QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxBUxXF+M9si6VJJg+6+JFvWLuluSQsl9Ula\n7e6/rLSzqOP8lbTM/3CyPvLGULL+yh3lx+qfOX9Lcttl/3xtsn7CLcV9px7HLu9x/q2S3jsR+g2S\nut19saTu7D6AKaRi+N39EUnvPfWskrQtu71N0mU59wWgzqp9zd/h7v3Z7dckdeTUD4AGqfkNPx97\n06DsGwdmts7Mes2sd1iHat0dgJxUG/4BM+uUpOz3YLkV3b3L3UvuXmpVW5W7A5C3asO/Q9La7PZa\nSffn0w6ARqkYfjO7U9Kjkj5hZnvN7CpJmyRdbGYvSvrj7D6AKaTidfvdfU2ZEgP2ORnZ/0ZN2w8f\nmFn1tp/8/LPJ+uu3tqQfYHSk6n2jWHzCDwiK8ANBEX4gKMIPBEX4gaAIPxAUU3RPA6df/0LZ2pVn\npkdk//Ok7mT9gs9enazPvfuxZB3NizM/EBThB4Ii/EBQhB8IivADQRF+ICjCDwTFOP80kJom+42v\nnJ7c9v92vJus33Dj9mT9b1Zfnqz7/36wbG3BPz2a3FYNnD4+Is78QFCEHwiK8ANBEX4gKMIPBEX4\ngaAIPxBUxSm688QU3c1n6E/PTdZv//o3kvVFM2ZVve9Pbr8mWV98W3+yfmRPX9X7nq7ynqIbwDRE\n+IGgCD8QFOEHgiL8QFCEHwiK8ANBVRznN7Mtki6VNOjuS7JlGyV9SdLr2Wob3P2BSjtjnH/q8fOW\nJuvHb9qbrN958o+q3vdpP/6zZP0T/1D+OgaSNPLinqr3PVXlPc6/VdLKCZZ/y92XZj8Vgw+guVQM\nv7s/ImmoAb0AaKBaXvNfa2a7zWyLmc3LrSMADVFt+G+VdLKkpZL6JX2z3Ipmts7Mes2sd1iHqtwd\ngLxVFX53H3D3EXcflXSbpGWJdbvcveTupVa1VdsngJxVFX4z6xx393JJT+fTDoBGqXjpbjO7U9KF\nkuab2V5JX5d0oZktleSS+iR9uY49AqgDvs+PmrR0nJCsv3rFqWVrPddvTm77gQpPTD//yopk/a3l\nbyTr0xHf5wdQEeEHgiL8QFCEHwiK8ANBEX4gKIb6UJjv7U1P0T3bZibrv/LDyfql115X/rHv60lu\nO1Ux1AegIsIPBEX4gaAIPxAU4QeCIvxAUIQfCKri9/kR2+jy9KW7X/5seoruJUv7ytYqjeNXcvPQ\nWcn67Pt7a3r86Y4zPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ExTj/NGelJcn6C19Lj7Xfdt62ZP38\nWenv1NfikA8n648NLUo/wGh/jt1MP5z5gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiCoiuP8ZrZA0nZJ\nHZJcUpe7bzazdkl3S1ooqU/Sanf/Zf1ajWvGopOS9Zev/GjZ2sYr7kpu+5nj9lfVUx42DJSS9Yc3\nn5Osz9uWvu4/0iZz5j8iab27nyHpHElXm9kZkm6Q1O3uiyV1Z/cBTBEVw+/u/e6+M7t9UNJzkk6U\ntErS0Y9/bZN0Wb2aBJC/Y3rNb2YLJZ0lqUdSh7sf/fzkaxp7WQBgiph0+M3sOEk/kHSdux8YX/Ox\nCf8mnPTPzNaZWa+Z9Q7rUE3NAsjPpMJvZq0aC/7t7n5vtnjAzDqzeqekwYm2dfcudy+5e6lVbXn0\nDCAHFcNvZibpO5Kec/ebxpV2SFqb3V4r6f782wNQL5P5Su95kr4g6Skz25Ut2yBpk6TvmdlVkn4u\naXV9Wpz6Ziz8vWT9rT/sTNav+McfJut//qF7k/V6Wt+fHo579N/LD+e1b/2f5LbzRhnKq6eK4Xf3\nn0oqN9/3Rfm2A6BR+IQfEBThB4Ii/EBQhB8IivADQRF+ICgu3T1JMzp/t2xtaMuc5LZfWfRwsr5m\n7kBVPeXhmn3Lk/Wdt6an6J7//aeT9faDjNU3K878QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxBUmHH+\nw3+Svkz04b8cStY3nPpA2dqK33mnqp7yMjDybtna+TvWJ7c97e9+lqy3v5kepx9NVtHMOPMDQRF+\nICjCDwRF+IGgCD8QFOEHgiL8QFBhxvn7Lkv/nXvhzHvqtu9b3jwlWd/88Ipk3UbKXTl9zGk3vlK2\ntnigJ7ntSLKK6YwzPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8EZe6eXsFsgaTtkjokuaQud99sZhsl\nfUnS69mqG9y9/JfeJR1v7X62Mas3UC893q0DPpT+YEhmMh/yOSJpvbvvNLO5kp4wswez2rfc/RvV\nNgqgOBXD7+79kvqz2wfN7DlJJ9a7MQD1dUyv+c1soaSzJB39zOi1ZrbbzLaY2bwy26wzs14z6x3W\noZqaBZCfSYffzI6T9ANJ17n7AUm3SjpZ0lKNPTP45kTbuXuXu5fcvdSqthxaBpCHSYXfzFo1Fvzb\n3f1eSXL3AXcfcfdRSbdJWla/NgHkrWL4zcwkfUfSc+5+07jlneNWu1xSerpWAE1lMu/2nyfpC5Ke\nMrNd2bINktaY2VKNDf/1SfpyXToEUBeTebf/p5ImGjdMjukDaG58wg8IivADQRF+ICjCDwRF+IGg\nCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxBUxUt357ozs9cl/XzcovmS9jesgWPTrL01\na18SvVUrz95OcvePTGbFhob/fTs363X3UmENJDRrb83al0Rv1SqqN572A0ERfiCoosPfVfD+U5q1\nt2btS6K3ahXSW6Gv+QEUp+gzP4CCFBJ+M1tpZs+b2UtmdkMRPZRjZn1m9pSZ7TKz3oJ72WJmg2b2\n9Lhl7Wb2oJm9mP2ecJq0gnrbaGb7smO3y8wuKai3BWb2YzN71syeMbO/yJYXeuwSfRVy3Br+tN/M\nWiS9IOliSXslPS5pjbs/29BGyjCzPkkldy98TNjMzpf0tqTt7r4kW/avkobcfVP2h3Oeu1/fJL1t\nlPR20TM3ZxPKdI6fWVrSZZK+qAKPXaKv1SrguBVx5l8m6SV33+PuhyXdJWlVAX00PXd/RNLQexav\nkrQtu71NY/95Gq5Mb03B3fvdfWd2+6CkozNLF3rsEn0VoojwnyjpF+Pu71VzTfntkh4ysyfMbF3R\nzUygI5s2XZJek9RRZDMTqDhzcyO9Z2bppjl21cx4nTfe8Hu/5e6+VNKnJV2dPb1tSj72mq2Zhmsm\nNXNzo0wws/RvFHnsqp3xOm9FhH+fpAXj7n8sW9YU3H1f9ntQ0n1qvtmHB45Okpr9Hiy4n99oppmb\nJ5pZWk1w7Jppxusiwv+4pMVmtsjMZkr6nKQdBfTxPmY2J3sjRmY2R9IKNd/swzskrc1ur5V0f4G9\n/JZmmbm53MzSKvjYNd2M1+7e8B9Jl2jsHf+XJf1tET2U6etkSU9mP88U3ZukOzX2NHBYY++NXCXp\nw5K6Jb0o6SFJ7U3U23clPSVpt8aC1llQb8s19pR+t6Rd2c8lRR+7RF+FHDc+4QcExRt+QFCEHwiK\n8ANBEX4gKMIPBEX4gaAIPxAU4QeC+n8DZI6NXofNrQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x21a4574c518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label:  5\n"
     ]
    }
   ],
   "source": [
    "plt.imshow(X_train[0])    # show first number in the dataset\n",
    "plt.show()\n",
    "print('Label: ', y_train[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADaVJREFUeJzt3X+MHPV5x/HPJ/b5iA9oMQTXNQ4ODUF1aHCki0kErRwR\nUiBBJkpCsVTLlShGLY2gitoiV1EttUopCkFuk0ZyghuDCNAGEFbipoJTWwuVOj6QsQHTmlCnsWt8\ngGltApxt/PSPG0cXuP3esb9mz8/7JZ1ud56ZnUfj+3hm97u7X0eEAOTzrrobAFAPwg8kRfiBpAg/\nkBThB5Ii/EBShB9IivADSRF+IKmZ3dzZLPfHSRro5i6BVN7QT3U4Rj2VdVsKv+3LJK2VNEPStyLi\nltL6J2lAF/qSVnYJoGBLDE153aYv+23PkPR1SZdLWiRpue1FzT4egO5q5Tn/EknPRcTzEXFY0r2S\nlrWnLQCd1kr450v6ybj7e6plP8f2KtvDtoePaLSF3QFop46/2h8R6yJiMCIG+9Tf6d0BmKJWwr9X\n0oJx98+qlgGYBloJ/1ZJ59p+n+1Zkq6RtLE9bQHotKaH+iLiqO0/kPRPGhvqWx8RT7etMwAd1dI4\nf0RskrSpTb0A6CLe3gskRfiBpAg/kBThB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkiL8\nQFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kRfiBpAg/kBThB5Ii\n/EBSLc3Sa3u3pEOS3pR0NCIG29EUgM5rKfyVj0fES214HABdxGU/kFSr4Q9Jj9h+3PaqdjQEoDta\nvey/OCL22j5T0sO2n42IzeNXqP5TWCVJJ2l2i7sD0C4tnfkjYm/1e0TSg5KWTLDOuogYjIjBPvW3\nsjsAbdR0+G0P2D7l+G1Jn5T0VLsaA9BZrVz2z5X0oO3jj/OdiPhBW7oC0HFNhz8inpd0QRt7AdBF\nDPUBSRF+ICnCDyRF+IGkCD+QFOEHkmrHp/pSePm6jzWsvXfFc8Vtnx2ZW6wfHu0r1uffU67P3vNq\nw9qxbc8Ut0VenPmBpAg/kBThB5Ii/EBShB9IivADSRF+ICnG+afoj//oOw1rnx14pbzxr7S486Xl\n8u6jrzWsrX3x4y3ufPr64cjZDWsDt/1CcduZQ4+3u52ew5kfSIrwA0kRfiApwg8kRfiBpAg/kBTh\nB5JyRHRtZ6d6TlzoS7q2v3b66ecubFh76UPl/0NP21k+xq/8qov1WR/632L91vMfaFi79N2vF7f9\n/msnF+ufmt34uwJa9XocLta3jA4U60tPOtL0vt///euL9Q+s2tr0Y9dpSwzpYBwo/0FVOPMDSRF+\nICnCDyRF+IGkCD+QFOEHkiL8QFKTfp7f9npJn5Y0EhHnV8vmSLpP0kJJuyVdHRGTfKh9ehv47pZC\nrbXHPrW1zfU3v7S0Ye0vLlpY3ve/luccuHXp+5voaGpmvn6sWB/Yvq9YP33z/cX6r81qPN/B7N3l\nuRAymMqZ/9uSLnvLspslDUXEuZKGqvsAppFJwx8RmyUdeMviZZI2VLc3SLqqzX0B6LBmn/PPjYjj\n12QvSCrPRwWg57T8gl+MfTig4ZvXba+yPWx7+IhGW90dgDZpNvz7bc+TpOr3SKMVI2JdRAxGxGCf\n+pvcHYB2azb8GyWtrG6vlPRQe9oB0C2Tht/2PZIek3Se7T22r5V0i6RLbe+S9InqPoBpZNJx/ohY\n3qA0PT+YfwI6+sL+hrWB+xvXJOnNSR574LsvN9FRe+z/3Y8V6x+cVf7z/cqB8xrWFv7d88Vtjxar\nJwbe4QckRfiBpAg/kBThB5Ii/EBShB9Iiim6UZuZZy8o1r+2+mvFep9nFOv/sPYTDWun73usuG0G\nnPmBpAg/kBThB5Ii/EBShB9IivADSRF+ICnG+VGbZ/9wfrH+kf7yTNNPHy5PPz7nmdfecU+ZcOYH\nkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaQY50dHjX7qIw1rT3zu9km2Ls/w9Hs33lisv/vffjjJ4+fG\nmR9IivADSRF+ICnCDyRF+IGkCD+QFOEHkpp0nN/2ekmfljQSEedXy9ZIuk7Si9VqqyNiU6eaxPT1\n35c3Pr+c7PI4/vL/urRYn/2DJ4v1KFYxlTP/tyVdNsHy2yNicfVD8IFpZtLwR8RmSQe60AuALmrl\nOf8XbG+3vd72aW3rCEBXNBv+b0g6R9JiSfsk3dZoRdurbA/bHj6i0SZ3B6Ddmgp/ROyPiDcj4pik\nb0paUlh3XUQMRsRg3yQf1ADQPU2F3/a8cXc/I+mp9rQDoFumMtR3j6Slks6wvUfSn0laanuxxkZT\ndku6voM9AuiAScMfEcsnWHxHB3rBNPSuU04p1lf8+qMNawePvVHcduTL5xTr/aNbi3WU8Q4/ICnC\nDyRF+IGkCD+QFOEHkiL8QFJ8dTdasmvNB4v1753xtw1ry3Z9trht/yaG8jqJMz+QFOEHkiL8QFKE\nH0iK8ANJEX4gKcIPJMU4P4r+77c/Wqxv/62/LtZ/dPRIw9qrf3VWcdt+7SvW0RrO/EBShB9IivAD\nSRF+ICnCDyRF+IGkCD+QFOP8yc2c/8vF+k1fuq9Y73f5T+iaJ1c0rL3nH/m8fp048wNJEX4gKcIP\nJEX4gaQIP5AU4QeSIvxAUpOO89teIOlOSXMlhaR1EbHW9hxJ90laKGm3pKsj4pXOtYpmeGb5n/iC\n7+0p1j9/8svF+t2HzizW536p8fnlWHFLdNpUzvxHJX0xIhZJ+qikG2wvknSzpKGIOFfSUHUfwDQx\nafgjYl9EPFHdPiRpp6T5kpZJ2lCttkHSVZ1qEkD7vaPn/LYXSvqwpC2S5kbE8e9ZekFjTwsATBNT\nDr/tkyXdL+mmiDg4vhYRobHXAybabpXtYdvDRzTaUrMA2mdK4bfdp7Hg3x0RD1SL99ueV9XnSRqZ\naNuIWBcRgxEx2Kf+dvQMoA0mDb9tS7pD0s6I+Oq40kZJK6vbKyU91P72AHTKVD7Se5GkFZJ22N5W\nLVst6RZJf2/7Wkk/lnR1Z1pESy44r1j+8zPvaunhv/7lzxfrv/jkYy09Pjpn0vBHxKOS3KB8SXvb\nAdAtvMMPSIrwA0kRfiApwg8kRfiBpAg/kBRf3X0CmLHoAw1rq+5t7b1Xi9bfUKwvvOvfW3p81Icz\nP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kxTj/CeDZ3z+tYe3K2Qcb1qbirH85XF4hJvz2NkwDnPmB\npAg/kBThB5Ii/EBShB9IivADSRF+ICnG+aeBN65cUqwPXXlboTq7vc3ghMGZH0iK8ANJEX4gKcIP\nJEX4gaQIP5AU4QeSmnSc3/YCSXdKmispJK2LiLW210i6TtKL1aqrI2JTpxrN7H8umlGsv3dm82P5\ndx86s1jvO1j+PD+f5p++pvImn6OSvhgRT9g+RdLjth+uardHxFc61x6ATpk0/BGxT9K+6vYh2zsl\nze90YwA66x0957e9UNKHJW2pFn3B9nbb621P+F1StlfZHrY9fESjLTULoH2mHH7bJ0u6X9JNEXFQ\n0jcknSNpscauDCZ8g3lErIuIwYgY7FN/G1oG0A5TCr/tPo0F/+6IeECSImJ/RLwZEcckfVNS+dMn\nAHrKpOG3bUl3SNoZEV8dt3zeuNU+I+mp9rcHoFOm8mr/RZJWSNphe1u1bLWk5bYXa2y0Z7ek6zvS\nIVryly8vKtYf+82FxXrs29HGbtBLpvJq/6OSPEGJMX1gGuMdfkBShB9IivADSRF+ICnCDyRF+IGk\nHF2cYvlUz4kLfUnX9gdksyWGdDAOTDQ0/zac+YGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gqa6O89t+\nUdKPxy06Q9JLXWvgnenV3nq1L4nemtXO3s6OiPdMZcWuhv9tO7eHI2KwtgYKerW3Xu1Lordm1dUb\nl/1AUoQfSKru8K+ref8lvdpbr/Yl0Vuzaumt1uf8AOpT95kfQE1qCb/ty2z/h+3nbN9cRw+N2N5t\ne4ftbbaHa+5lve0R20+NWzbH9sO2d1W/J5wmrabe1tjeWx27bbavqKm3Bbb/2fYztp+2fWO1vNZj\nV+irluPW9ct+2zMk/aekSyXtkbRV0vKIeKarjTRge7ekwYiofUzY9m9IelXSnRFxfrXsVkkHIuKW\n6j/O0yLiT3qktzWSXq175uZqQpl542eWlnSVpN9Rjceu0NfVquG41XHmXyLpuYh4PiIOS7pX0rIa\n+uh5EbFZ0oG3LF4maUN1e4PG/ni6rkFvPSEi9kXEE9XtQ5KOzyxd67Er9FWLOsI/X9JPxt3fo96a\n8jskPWL7cdur6m5mAnOradMl6QVJc+tsZgKTztzcTW+ZWbpnjl0zM163Gy/4vd3FEbFY0uWSbqgu\nb3tSjD1n66XhminN3NwtE8ws/TN1HrtmZ7xutzrCv1fSgnH3z6qW9YSI2Fv9HpH0oHpv9uH9xydJ\nrX6P1NzPz/TSzM0TzSytHjh2vTTjdR3h3yrpXNvvsz1L0jWSNtbQx9vYHqheiJHtAUmfVO/NPrxR\n0srq9kpJD9XYy8/plZmbG80srZqPXc/NeB0RXf+RdIXGXvH/kaQ/raOHBn2dI+nJ6ufpunuTdI/G\nLgOPaOy1kWslnS5pSNIuSY9ImtNDvd0laYek7RoL2ryaertYY5f02yVtq36uqPvYFfqq5bjxDj8g\nKV7wA5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+Q1P8DC8wZVCobNIoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x21a45762550>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label:  7\n"
     ]
    }
   ],
   "source": [
    "plt.imshow(X_test[0])    # show first number in the dataset\n",
    "plt.show()\n",
    "print('Label: ', y_test[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# reshaping X data: (n, 28, 28) => (n, 784)\n",
    "X_train = X_train.reshape((X_train.shape[0], -1))\n",
    "X_test = X_test.reshape((X_test.shape[0], -1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# use only 33% of training data to expedite the training process\n",
    "X_train, _ , y_train, _ = train_test_split(X_train, y_train, test_size = 0.67, random_state = 7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# converting y data into categorical (one-hot encoding)\n",
    "y_train = to_categorical(y_train)\n",
    "y_test = to_categorical(y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(19800, 784) (10000, 784) (19800, 10) (10000, 10)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic MLP model\n",
    "- Naive MLP model without any alterations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Activation, Dense\n",
    "from keras import optimizers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = Sequential()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.add(Dense(50, input_shape = (784, )))\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(50))\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(50))\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(50))\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(10))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sgd = optimizers.SGD(lr = 0.001)\n",
    "model.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = ['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "history = model.fit(X_train, y_train, batch_size = 256, validation_split = 0.3, epochs = 100, verbose = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8XNV9///XR6PNkrVL3iTZksH7bmRjMDsmGEhYAgmQ\npSGNQyAhxG3SFNomhKRJ0/4IJekX4hKWfBsglK/ZHGIgYTGEgI1tjLG877Zk7bK20Tqaz++POxqP\n5JE0ljWSPPN5Ph5+aObcc0fnKOGto3PPPVdUFWOMMdEjZrgbYIwxZmhZ8BtjTJSx4DfGmChjwW+M\nMVHGgt8YY6KMBb8xxkQZC35jjIkyFvzGGBNlLPiNMSbKxA53A4LJzs7WgoKC4W6GMcacMTZv3lyt\nqjmh1A0p+EVkOfBLwAU8pqo/73H8i8A/AgI0Aneq6taA4y5gE1Cqqp/u7/sVFBSwadOmUJpmjDEG\nEJHDodbtd6rHF9oPA1cBM4FbRWRmj2oHgYtVdQ7wE+DRHse/A+wMtVHGGGPCJ5Q5/sXAPlU9oKrt\nwLPAdYEVVPV9VT3ue7seyOs6JiJ5wDXAY4PTZGOMMacjlODPBY4GvC/xlfXma8CrAe8fAr4PePv6\nJiJyu4hsEpFNVVVVITTLGGPMQAzqxV0RuRQn+C/wvf80UKmqm0Xkkr7OVdVH8U0RFRUVnbRXdEdH\nByUlJbS2tg5mk6NWYmIieXl5xMXFDXdTjDFDLJTgLwXyA97n+cq6EZG5ONM5V6lqja94KXCtiFwN\nJAKpIvKUqn7pVBtaUlJCSkoKBQUFiMipnm4CqCo1NTWUlJRQWFg43M0xxgyxUKZ6NgJTRKRQROKB\nW4A1gRVEZCLwAvBlVd3TVa6q96pqnqoW+M57ayChD9Da2kpWVpaF/iAQEbKysuyvJ2OiVL8jflX1\niMhdwOs4yzmfUNXtInKH7/gq4IdAFvCIL5g9qlo02I210B889rM0JnqFNMevqmuBtT3KVgW8XgGs\n6Ocz1gHrTrmFxhhzpnHXwK5XYOJ5kDN1uFtzEtuyIUR1dXU88sgjp3ze1VdfTV1dXZ91fvjDH/LG\nG28MtGnGmJFi/9vw7BfhF9PgD3fD2/863C0KyoI/RL0Fv8fj6fO8tWvXkp6e3medH//4xyxbtuy0\n2meMGWZbnobfXQ9HP4RzvwETz4eK7cPdqqAs+EN0zz33sH//fubPn8+iRYu48MILufbaa5k507mJ\n+frrr+ecc85h1qxZPProiRuXCwoKqK6u5tChQ8yYMYOvf/3rzJo1i0996lO0tLQAcNttt7F69Wp/\n/fvuu4+FCxcyZ84cdu3aBUBVVRVXXHEFs2bNYsWKFUyaNInq6uoh/ikYY4LathrW3AWTL4WV2+DK\nn8Lki6FmP7S7ez3twT/t5vurt/Z6PFxG5CZt/bn/D9vZcaxhUD9z5oRU7vvMrF6P//znP6e4uJiP\nP/6YdevWcc0111BcXOxfDvnEE0+QmZlJS0sLixYt4sYbbyQrK6vbZ+zdu5ff//73/OY3v+Hzn/88\nzz//PF/60smLnLKzs/noo4945JFHeOCBB3jssce4//77ueyyy7j33nt57bXXePzxxwe1/8aYAdr5\nCrxwuzOff8szEJfolI+dDShU7oS8k9e6HK1t5pF1+3HFCD+7YQ6xrqEbh9uIf4AWL17cbQ38r371\nK+bNm8eSJUs4evQoe/fuPemcwsJC5s+fD8A555zDoUOHgn72Zz/72ZPqvPfee9xyyy0ALF++nIyM\njEHsjTFmQMqLYfXfwoQF8IX/hfikE8fG+gaSFcVBT/0/b+3D41XaPF72V/X+V0E4nJEj/r5G5kMl\nOTnZ/3rdunW88cYbfPDBByQlJXHJJZcEXSOfkJDgf+1yufxTPb3Vc7lc/V5DMMYMk44WeH4FjEp3\nQj8hpfvx9EkQn+L8cujhcI2b1R+VcOGUbP6yt5ri0nqmjUs5qV642Ig/RCkpKTQ2NgY9Vl9fT0ZG\nBklJSezatYv169cP+vdfunQpzz33HAB/+tOfOH78eD9nGGPC6s/3QdVOuP4RSM4++XhMDIydGfQC\n73+9tY/YGOE/bppLYlwM2wd56ro/Z+SIfzhkZWWxdOlSZs+ezahRoxg7dqz/2PLly1m1ahUzZsxg\n2rRpLFmyZNC//3333cett97K7373O8477zzGjRtHSsrQjRCMMQH2vgEf/jeceyec3ceKvLGznQu/\nquC7afJgtZsXt5Ty1fMLGJ82ihnjUyk+Vj9EDXeI6kn7oQ27oqIi7fkglp07dzJjxoxhatHwa2tr\nw+VyERsbywcffMCdd97Jxx9/fFqfGe0/U2MGpKMFfjkPkrLg62+fuJgbzMbH4Y9/Dyu3oWn5fHCg\nhs7nvkp9i4dz/+ElclIS+JeXtvHylmNsve9TxMQM/I56Edkc6o4JNuI/Qxw5coTPf/7zeL1e4uPj\n+c1vfjPcTTImOtUehKYKuPJnfYc+wLg5AGzd9B73bs+nquww6xPfwxuXSFySC4DZE9J4av0RjtQ2\nU5Cd3NenDRoL/jPElClT2LJly3A3wxjjrnS+pozrv+4Y5y/qN9a9RVvGl/jNgoO4dnpxdTY7q30m\nzGd2bhoA2481DFnw28VdY4w5FU2+B0Ulj+m36hv7mzmsYzk/uYxXvn0h82tfd1b7ABzdAMCUsaOJ\njZEhnee34DfGmFPRNeIfndNntb/uq+abz3zEsYTJnJtcxqjju6BiG5x3F6TmwZEPAEiIdTF1bArF\npRb8xhgzMjVVgiseEnvfg8vd5uFbz3xEYVYy84suJKb2AGx6EmJiYfaNMPFcOLLBWe0DzJqQyo5j\nDQzVYhsLfmOMORXuakjO8S/PDOb3Hx6hrrmDf7txDqPy5wIKm38LZ18ByVmQvwQaj0G98zjz2blp\n1LjbKW8YmocjWfCHyejRowE4duwYN910U9A6l1xyCT2Xrfb00EMP0dzc7H8fyjbPxpgwclc6wd+L\ndo+Xx/5ykHMLM1k4McO3Zw/g7YB5NzuvJ/ru9Tni3Ow5OzcVgO2lQ3MjlwV/mE2YMMG/8+ZA9Az+\nULZ5NsaEUVMljO79wu5LH5dS3tDKnZec5RR0bd2QkAZTr3LKxs5yynzBP31cKiIM2QVeC/4Q3XPP\nPTz88MP+9z/60Y/413/9Vy6//HL/Fsovv/zySecdOnSI2bOd3/gtLS3ccsstzJgxgxtuuKHbXj13\n3nknRUVFzJo1i/vuuw9wNn47duwYl156KZdeeilwYptngAcffJDZs2cze/ZsHnroIf/36237Z2PM\nIHBX9bqix+tVVr2znxnjU7l4qu+vgpgYmP8FWHr3iXX/MS7IX+QP/uSEWCZnJw/Z1g0hreMXkeXA\nL3GeufuYqv68x/EvAv8ICNAI3KmqW0UkH/gfYCygwKOq+svTbvWr90D5ttP+mG7GzYGrft7r4Ztv\nvpmVK1fyrW99C4DnnnuO119/nbvvvpvU1FSqq6tZsmQJ1157ba/Ps/31r39NUlISO3fu5JNPPmHh\nwoX+Yz/96U/JzMyks7OTyy+/nE8++YS7776bBx98kLfffpvs7O57gWzevJknn3ySDRs2oKqce+65\nXHzxxWRkZIS8/bMx5hSpOsHfy4qeP+2o4ECVm1/duqB7Dlz9HydXzl8C6/4NWupgVDqzJqSx6VBt\nmBreXb8jfhFxAQ8DVwEzgVtFZGaPageBi1V1DvAToOtJJB7gu6o6E1gCfCvIuWeEBQsWUFlZybFj\nx9i6dSsZGRmMGzeOf/qnf2Lu3LksW7aM0tJSKioqev2Md9991x/Ac+fOZe7cuf5jzz33HAsXLmTB\nggVs376dHTt29Nme9957jxtuuIHk5GRGjx7NZz/7Wf7yl78AoW//bIw5Ra110Nne6xz/f7+7n4mZ\nSVw9O4SbuyaeCyiUONf5bl6Uz8plU4dkZU8oI/7FwD5VPQAgIs8C1wH+ZFLV9wPqrwfyfOVlQJnv\ndaOI7ARyA88dkD5G5uH0uc99jtWrV1NeXs7NN9/M008/TVVVFZs3byYuLo6CgoKg2zH35+DBgzzw\nwANs3LiRjIwMbrvttgF9TpdQt382xpyiPm7eamjtYMuROr57xdTQHqqSWwTictbzT1nG0rOD7PAZ\nJqHM8ecCRwPel/jKevM14NWehSJSACwANgQ7SURuF5FNIrKpqqoqhGYNvZtvvplnn32W1atX87nP\nfY76+nrGjBlDXFwcb7/9NocPH+7z/IsuuohnnnkGgOLiYj755BMAGhoaSE5OJi0tjYqKCl599cSP\nr7ftoC+88EJeeuklmpubcbvdvPjii1x44YWD2FtjzEn6uHlrb4Xz3+nMCamhfVbCaGeK2Xcj11Aa\n1L16RORSnOC/oEf5aOB5YKWqBr16oaqP4psiKioqGnlbhgKzZs2isbGR3Nxcxo8fzxe/+EU+85nP\nMGfOHIqKipg+fXqf599555189atfZcaMGcyYMYNzzjkHgHnz5rFgwQKmT59Ofn4+S5cu9Z9z++23\ns3z5ciZMmMDbb7/tL1+4cCG33XYbixcvBmDFihUsWLDApnWMCacmX/AHGfHvKneC/5QeqHLWZfDX\nX/ruDRi6EX+/2zKLyHnAj1T1St/7ewFU9d961JsLvAhcpap7AsrjgFeA11X1wVAaZdsyDw37mRpz\nijb8N7z6ffiH/ScF9X0vF7N6cwnF91/Z6wKPk5QXw6qlcM2DsOhrp9W0U9mWOZSpno3AFBEpFJF4\n4BZgTY9vOBF4Afhyj9AX4HFgZ6ihb4wxI1ZTpTMvPyrzpEO7KxqZOi4l9NAHZz1/9jQofn4QG9m/\nfoNfVT3AXcDrwE7gOVXdLiJ3iMgdvmo/BLKAR0TkYxHpGq4vBb4MXOYr/1hErh78bhhjzBBwVzoj\n/Zju0amq7C5vZPqpPjdXBObcBIffh/rSQWxo30Ka41fVtcDaHmWrAl6vAFYEOe89nLX9g0JVT+23\nqenVSHzymjEjXlPwm7eqmto43tzB1LEDeBzqrM/C2z+FHS/Bed8ahEb274y5czcxMZGamhoLrEGg\nqtTU1JCY2M/Tg4wx3bkrg67o2T2QC7tdss+G8fOGdLrnjHkCV15eHiUlJYzUpZ5nmsTERPLy8oa7\nGcacWZqqIGvKScX+4B/IiB+crZr//EOoPQCZk0+nhSE5Y4I/Li6OwsLC4W6GMSZaqfY64t9T0Uj2\n6ASyRicEOTEEsz7rBH/xC3DR906zof07Y6Z6jDFmWLU1gqc16Bz/7vJGpo0bPfDPTs939u4pfuE0\nGhg6C35jjAmF2zfN3GNLZq9X2VPRNLALu4EWfQ3Ovhw6O07vc0Jwxkz1GGPMsPLftdt9qufo8WZa\nOjpPfSlnT3M/7/wbAjbiN8aYUPj36ek+4u+6sHvaI/4hZMFvjDGh6GWfnj0VFvzGGBOZ3FWAQFJW\nt+Jd5Y3kZ44iOeHMmTm34DfGmFA0VTqh7+oe8HsqGge+fn+YWPAbY0wo3FUnze9XN7VxoMo9sDt2\nh5EFvzHGhMJd1W0rZlXlBy8VEyPCDQv6ejbVyGPBb4wxoWiq7HZh95VPyni1uJyVV0zh7DE24jfG\nmMgTMNVT1djGD18uZl5+OrdfGP69dQabBb8xJjps/V/4/RecPXdOVXsztDdBco5/isfd1skDN80N\n7cHqI8yZ12JjjBmInWtg9x+h7vCpn+u7ecubnMMPXi7mte3l/P2npjLlDFvN08WC3xgTHap2O18P\n/fXUzy3bCsCvtnh5av0RvnHxZL5x0Zk3xdPlzLnjwBhjBsrT5ux1D3DoPVjwxT6re73KwRo3eysa\nqWvuoGjT0+TEpPFfezO456rp3HHxWUPQ6PAJKfhFZDnwS8AFPKaqP+9x/IvAP+I8ZrERuFNVt4Zy\nrjHGhF3NftBOiEtygr8Xeysa+eeXiikurae5vROABNrZnPAOf9Sl/OzG+dy8aOJQtTps+g1+EXEB\nDwNXACXARhFZo6o7AqodBC5W1eMichXwKHBuiOcaY0x4Ve1yvs65CT76Hzh+GDImdavibvPwjac2\nU9/cweeL8pk5IZUZ41LJrXiL0X9o5fNfvgs5+8wPfQhtxL8Y2KeqBwBE5FngOsAf3qr6fkD99UBe\nqOcaY0zYVe0GiYFzbnOC//BfuwW/qvIvLxVzqNrN0yuWcN5ZAfvxfPgqjMpACi8c+naHSSgXd3OB\nowHvS3xlvfka8Oqpnisit4vIJhHZZM/VNcYMqqpdkFEI4xfAqEza9r3D2m1llNa1APD/NpXw4pZS\nVi6b2j30PW2w+1WYfg244oap8YNvUC/uisilOMF/wameq6qP4kwRUVRUNICFtsYY04uq3ZAzHWJi\noGAprXvf5ZubPgJgYmYSlY2tXHB2Nt+69Ozu5x1YB20NMPOGoW9zGIUy4i8F8gPe5/nKuhGRucBj\nwHWqWnMq5xpjTNh0dkDNPsiZ5rwvuJC0tmMsSG3gh5+eybRxKcwYn8p/3jwfV4x0P3fHy5CYBoUX\nDX27wyiUEf9GYIqIFOKE9i3AFwIriMhE4AXgy6q651TONcaYsKo9CN4OZ8QPMGkpAMtHH+BvL7iV\nv72gMPh5nnbY9QpMuwZi44eosUOj3+BXVY+I3AW8jrMk8wlV3S4id/iOrwJ+CGQBj4gIgEdVi3o7\nN0x9McaYk3Wt6MmZCoCOmUE9oymijyg6/D68+WNorYdZ1w9BI4dWSHP8qroWWNujbFXA6xXAilDP\nNcaYIdN1x262E/x1LZ1s6JzB0uYtzr49EjC901QJL90J+96A0ePgmgdhyqeGodHhZVs2GGMiW9Uu\nSJ8I8ckAHK5t5k3vAlJaj8HOP3Sv+9q9cPAvcMVP4O4tsOhr3X8xRAgLfmNMZOta0eNzuMbNC50X\n0pY5HV7/Z+ho8R14H4pXwwUrYendEJ80TA0OPwt+Y0zk8nZC9Z4TK3qAQ9XNdOJCrv4PqD8Cf/2V\nU+/V70NqHixdOYwNHhq2SZsxJnIdPwSdbd1H/LVuxqclEn/2xTDrBnjvQadO+Tb43G8jeqTfxUb8\nxpjIsuUp+MN3nN04uy7sdpvqaWZipi/cr/gJIPCXX8CkC2Bm5K3gCcaC3xgTWT58FDb/Fv7PInjz\nfqfMt6IHnOAvyHIu9JKeDxf/A8QmwlX/HpEXcoOxqR5jTORQdbZgnn0jjMp0fgFkFEBiKgBNbR6q\nm9qYmBUwnXPhd2HRCucO3ShhwW+MiRyN5c6zcSeeB4u/7qzQ6ezwHz5S0wxwYsTfJYpCHyz4jTGR\npGav8zXLt9laWl63w4dr3ABMyor8C7h9sTl+Y0zkqNnnfM2eEvTw4VpnxG/Bb4wxkaJ6H8SOgpQJ\nAPx5RwX/9eZe/+HDNW6ykuNJSYycvfUHwoLfGBM5avY60zwxTrT978aj/OLPe9hxrAHwLeWM8tE+\nWPAbYyJJ9V7IPvEwlfIGZzuGX7+zH+ixlDOKWfAbYyKDpx3qDp+4sAuU17cSGyP88ZNj7C5v5Fh9\ny4mbt6KYBb8xJjIcPwjqhSznwm6bp5PqpnZuXTyRWFcMP3ipGFUoyLbgt+WcxpjIUO27iOub6qls\naANgTm4aivLU+iMATMy0qR4b8RtjIkOPNfzlDa0AjEtL5BsXneV/nm6BXdy14DfGRIiafZA8xn8X\nblm9E/zj0xLJz0zihgW55KQkkJkcWc/PHYiQgl9ElovIbhHZJyL3BDk+XUQ+EJE2Eflej2N/JyLb\nRaRYRH4vIomD1XhjjPGr3tftxq2K+hMjfoCf3jCbP377AiRKNmLrS7/BLyIu4GHgKmAmcKuIzOxR\nrRa4G3igx7m5vvIiVZ2N88D1Wwah3cYY013NXsg6y/+2rL6V5HiX/2athFgXY1Jt3AmhjfgXA/tU\n9YCqtgPPAtcFVlDVSlXdCHQEOT8WGCUisUAScOw022yMMd0110JzjX9FDzhr+LtG+6a7UII/Fzga\n8L7EV9YvVS3F+SvgCFAG1Kvqn061kcYY06ca5watwKmesvpWxqeNGqYGjWxhvbgrIhk4fx0UAhOA\nZBH5Ui91bxeRTSKyqaqqKpzNMsZEmp67cuLM8Y+1qZ2gQgn+UiA/4H2erywUy4CDqlqlqh3AC8D5\nwSqq6qOqWqSqRTk5OSF+vDHG4Kzhj4l1HroCdHqVisY2xttUT1ChBP9GYIqIFIpIPM7F2TUhfv4R\nYImIJIlzKf1yYOfAmmqMMb2o2eeEvsu5kFvd1EanV22Ovxf93rmrqh4RuQt4HWdVzhOqul1E7vAd\nXyUi44BNQCrgFZGVwExV3SAiq4GPAA+wBXg0TH0xxkSr2gOQ2X1FD2Aj/l6EtGWDqq4F1vYoWxXw\nuhxnCijYufcB951GG40xUUpVueuZLdy6eCIXTMnurRLUHYFJJ2aRy33Bb3P8wdmdu8aYEavN4+WP\n28p4d28fCz5a66CtAdIn+YvK653tmG3EH5wFvzFmxHK3eQCodbf3XqnO2XyN9In+orKGVuJdMbY9\nQy8s+I0xI1ZzeycAdc19BP/xw87XgOAvr29lbFqCbc/QCwt+Y8yI5W4f2Ii/vL6V8al281ZvLPiN\nMSPWiRF/sN1gfOqOQHwKjMrwF5U3tNpSzj5Y8BtjRqzmNif4a/ua6qk7AhmTwDeto6q+7Ros+Htj\nwW+MGbG6pnrqWzro9GrwSnVHuk3zHG/uoN3jtaWcfbDgN8aMWM2+4Fd1wv8kXWv4e8zvgy3l7IsF\nvzFmxOqa44deLvC2HIf2xu7B3+Cs4bc5/t5Z8BtjRqyuOX7oZUln3clLOct6PHnLnMyC3xgzYnXN\n8UMvI37/Us4Td+1W1LcSI5AzOiHczTtjWfAbY0aswKme40FH/EHu2q1vZUxKIrEui7fe2E/GGDPy\nlG6Gjlaa2z2MinMBzmqdQF954kM2b90KCWkwKt1ffqDaTW6G3bzVFwt+Y8zI4q6Bx5bB1mdobusk\na3Q88bExHA+Y6lFVNh6qpbnyAJp+4jlRDa0dfHy0jiWTM4ej5WcMC35jzMjSVA7qhbqjuNs9jE6I\nJSMprttUT0OLh+b2TsZ0VtCYOMFf/v6+ajq9ykVT7Cl+fbHgN8aMLO5q52tTJc3tnSTFu8hIiqfW\nfWKq51h9C6DkSRUHPVn+8nf2VDM6IZaFkzIwvbPgN8aMLM2+4Hd3BX8smcnxzoj/3QfgwDuU1beQ\nQSPJ0sZHjWmAM/3z7p4qzj8rizi7sNsn++kYY0YWd43ztakCd5vHP+Kvc7fCun+Dl75JRW09eeL8\nglhfk0xDawcHqt2U1rVw0VSb5ulPSMEvIstFZLeI7BORe4Icny4iH4hIm4h8r8exdBFZLSK7RGSn\niJw3WI03xkSg5u5TPckJsWQkxzlTQF4PNJQwbvdTTIxx6h3xZvPXvdW8s9t5StfFFvz96veZuyLi\nAh4GrgBKgI0iskZVdwRUqwXuBq4P8hG/BF5T1ZtEJB5IOv1mG2MiVtccv7uKFmn3j/iT2iohHkhI\nY3HJkxxO/BR0Qn3CONbtrqKisZXC7GTyMy1i+hPKiH8xsE9VD6hqO/AscF1gBVWtVNWNQLeFtiKS\nBlwEPO6r166qdYPScmNMZGr2TfV4PcR11PuDP4fjTvkVPyK5s4FbO/8AiWnMnzKJt3ZXsv5ADRf1\n9kB2000owZ8LHA14X+IrC0UhUAU8KSJbROQxEUk+xTYaY6JJV/ADyR3H/Rd3x4kv+KdcyZ9dF5FA\nG6RP4pKpY6hqbKO1w8vF02yaJxThvrgbCywEfq2qCwA3cNI1AgARuV1ENonIpqqqqjA3yxgzYrmr\n/U/TypE6khNcpCfFMVaOowianMPP226kU2IhfaI/7ONdMSyZnNXXJxufUIK/FMgPeJ/nKwtFCVCi\nqht871fj/CI4iao+qqpFqlqUk2O/tY2JWs3VMGYWANnU+0f8Y6mlPTGb422w35PDm/Megkv/ibGp\niczPT+f8s7NIiu/3sqUhhIu7wEZgiogU4gT+LcAXQvlwVS0XkaMiMk1VdwOXAzv6O88YE6W8Xmiu\nhTEz4PB75Eidf45/rBzHnZDDsTpnv33v2ctg7HgAfvvVRcTEyHC2/IzSb/CrqkdE7gJeB1zAE6q6\nXUTu8B1fJSLjgE1AKuAVkZXATFVtAL4NPO1b0XMA+GqY+mKMOdO11oF2QtZZeGPiyZEGkuJjyfDN\n8TfEFvj32x+fdmIjtvSk+OFq8RkppL+LVHUtsLZH2aqA1+U4U0DBzv0YKDqNNhpjokXXUs6kbDpG\nZZPT4czxJ8e7GCvHOeQ6h7J6Z8Rvj1YcOLtz1xgzcnSt6EnKpC0hmxzqSIqPRTrbyZRGqsmkrL6V\nOJeQbQ9aGTALfmPMyNF1125yNq0JWWRLA0nxLmgsB6DMm05ZXQtjUxNtTv80WPAbY0aOgKked1yW\ns5wzPhYaywAo6UznWH0rE9LsQSunw4LfGDNyBIz4G2MzyKSBpDj8wX+oLZWy+hbGp9v8/umwRa/G\nmJHDXQPxKRCbQL0rE5coyZ56aHCCf29LCuXtrd1W9JhTZ8FvjBk5mqsh2bn7ti7GuXs3sb0aGsvw\nSDyHWhIAtRU9p8mmeowxI4e7GpKcjdZqcR6gLk2V0FhGc2IO4FzQteA/PRb8xpiRo7kGkp3gr/IF\nP+4qaCynY9QYf7UJ6TbVczos+I0xI0dzDSQ5Uz2V3lSnrKkCGo7hGT3eX81G/KfHgt8YMzKo+qZ6\nnOA/7omnlQRoqoTGcmJSnOBPiI0hM9m2aDgdFvzGmJGhvQk62/xTPc0dndS7MqBmH3S4iU13HgMy\nPi0REbt563RY8BtjRoaAm7cA3G2dNLgyoWwrAImZEwAYZ9M8p82C3xgzMnTt09M14m/34I7L8N+8\nlZiZR5xL7K7dQWDr+I0xI0OPEX9zeyfuxCznuX2ApE7gaxd0sGRy5jA1MHJY8BtjRgb/iN+5uNvc\n3klbWsDT+FLGcc9V9sjuwWBTPcaYkaG55xy/h/ZE3zN0E9Ig3kJ/sFjwG2NGBnc1uBIgPplOr9Lm\n8eLpumkrdXzf55pTYsFvjBkZuu7aFaG53QOA13ehl5Rxw9iwyBNS8IvIchHZLSL7ROSeIMeni8gH\nItImIt/KQ6MtAAAR/0lEQVQLctwlIltE5JXBaLQxJgIF3LzV3N4JgCaPdY6lTBiuVkWkfoNfRFzA\nw8BVwEzgVhGZ2aNaLXA38EAvH/MdYOdptNMYE+maq/1LOd1tzojflTIGEEi14B9MoYz4FwP7VPWA\nqrYDzwLXBVZQ1UpV3Qh09DxZRPKAa4DHBqG9xphIFbAzZ9eIPyEpBW7+HSxaMZwtizihLOfMBY4G\nvC8Bzj2F7/EQ8H0g5RTOMcZEm4CdObuCPyneBWd/ZjhbFZHCenFXRD4NVKrq5hDq3i4im0RkU1VV\nVTibZYwZaTpanb16fHP8bt/F3aR413C2KmKFEvylQH7A+zxfWSiWAteKyCGcKaLLROSpYBVV9VFV\nLVLVopycnGBVjDGRqud2DW3OiD85we4xDYdQgn8jMEVECkUkHrgFWBPKh6vqvaqap6oFvvPeUtUv\nDbi1xpjIVO+bTfat3uka8Y+KsxF/OPT761RVPSJyF/A64AKeUNXtInKH7/gqERkHbAJSAa+IrARm\nqmpDGNtujIkUVbucrznTAGhptxF/OIX0U1XVtcDaHmWrAl6X40wB9fUZ64B1p9xCY0zkq9oDcUmQ\n5swq2xx/eNmdu8aY4Ve1C7LOhhgnkprbOnHFCAmxFlHhYD9VY8zwq94DOdP9b93tHpLiXPakrTCx\n4DfGDK+2Jufibs5Uf1FLeydJCTbNEy4W/MaY4VW9x/nabcTfSXK8XdgNFwt+Y8zwqtrtfM2e5i9q\nbvPYiD+MLPiNMcOrejfExEFmob/ImeO3EX+4WPAbY8KqrL6FH63Zzvv7q1HVkytU7Yass8AV5y+y\nOf7wsuA3xoTVHz8p47fvH+ILv9nAsgff4XfrD3f/BVC1G2/2NJ7860Gqm9oAm+MPNwt+Y0xYHax2\nkzYqjgc+N4/RCbH84KViXv74mHOwoxWOH6S4fRz3/2EH//D/tqKqzhy/3bwVNhb8xpiwOlzTTEF2\nMjedk8eL31zK9HEp/OrNvXg6vVC7H9TL6iPJpCTE8vbuKl7cUoq7vdOCP4ws+I0xYXWoxk1hVhIA\nMTHCymVTOVDtdkb9vj16Njbl8Mtb53POpAzu/8MO3G0ekmyfnrCx4DfGhE2bp5NjdS1Mykr2l105\naywzx6fyq7f24qnYRScxJE+YxqXTxvDvN86lpaMTj1dJthF/2FjwG2PC5mhtC16Fguwkf5mI8HdX\nTOVwTTObN6/nqDeHb1w+GxHh7DGj+c7lUwBIsou7YWM/WWNMUD9as53c9FF8/aLJIdU/WtvM9mP1\nLJ893l92uMYNQEHAiB9g2YwxzMlNI63qIOUJk1g2Y4z/2DcumozXq3xq1thB6IUJxkb8xpiTeDq9\nPLvxCM9/VBLyOT9bu5M7n/6IxtYOf9nB6uDBLyJ8b9lkJssxxk6e220ztlhXDN++fAp5GUmY8LDg\nN8acZH+Vm9YOL3srm2jt6Oy3fkNrB2/uqkQVtpXW+8sP1zSTmhhLelLcSedcnOMmXjopnHHOoLbd\n9M+C3xhzkk9K6gDo9Co7y/p/kN5r28pp93gB+Phonb/8UI2bwuzk4Nsrl2x0vo6bc/oNNqfEgt8Y\nc5JtpfXExjhhXRwwgu/Ni1tKKchKYlJWElt7BP+kHtM8fgfWQVI2jJk1GE02pyCk4BeR5SKyW0T2\nicg9QY5PF5EPRKRNRL4XUJ4vIm+LyA4R2S4i3xnMxhtjwmNbaT0LJ2aQmRzfbeommPL6VtYfrOG6\n+bnMz09n61GnfrvHS+nxFgqygszVq8KBd6DwIv9Tt8zQ6fcnLiIu4GHgKmAmcKuIzOxRrRa4G3ig\nR7kH+K6qzgSWAN8Kcq4xZgTxdHrZcayBOXlpzM5NY1tp31M9a7aWogrXL3CCv7yhlfL6VkqON/uW\ncgYZ8VfthqZymHxJWPpg+hbKr9rFwD5VPaCq7cCzwHWBFVS1UlU3Ah09ystU9SPf60ZgJ5A7KC03\nxoTF3som2jxe5ualMSc3lb0VjX1e4H1xyzHm5adTmJ3MvPx0wJnnP+Rbyhl0qufAOufr5EsGt/Em\nJKEEfy5wNOB9CQMIbxEpABYAG071XGPM0NlW4kzVzM5NY05uGh6vsqu8MWjdPRWN7Cxr4Pr5EwCY\nOT6VOJewtaSOQ9XNAMGneg6+AxkFkDEpLH0wfRuSyTURGQ08D6xU1aB/N4rI7SKySUQ2VVVVDUWz\njDFBbCutZ3RCLIVZyczOTQN6v8D72/cP4YoRPj3XCf7EOBczxqfy8ZE6Dte4SUmMJTM5vvtJnR44\n9J6N9odRKMFfCuQHvM/zlYVEROJwQv9pVX2ht3qq+qiqFqlqUU5OTqgfb4wZZJ+U1jNrQioxMUJu\n+ijSk+KCBv/v1h/mmQ1H+PKSSeSkJPjL5+Wls620ngPVbgqygizlPLYF2hqg8OJwd8X0IpTg3whM\nEZFCEYkHbgHWhPLh4vwv/jiwU1UfHHgzjTFDoaPTy86yBubmOSN9EWFObtpJK3ve3FnBfS8Xc9n0\nMfzLNTO6HZuXn05Tm4cNB2uDX9jtmt+34B82/Qa/qnqAu4DXcS7OPqeq20XkDhG5A0BExolICfD3\nwL+ISImIpAJLgS8Dl4nIx75/V4etN8aY07KnopF2j5c5een+stm5aeypaKTN41zg3VZSz13PbGHm\nhFT+69YFxLq6x8j8fOeXRrvH2/v8/rg5kJwVvo6YPoW0SZuqrgXW9ihbFfC6HGcKqKf3gCC37Blj\nRqKuC7tzfHP7Xa87OpXd5Y2owt888SGZyfE88ZVFJAfZM39y9miKEkq4pvNNMtJ+0v1gezMc3QDn\nfiOs/TB9s905jTF+20rrSUmMZVLmiZF61y+B//ngMK8Vl5ORHMczK5YwJjUx6GfECPws4Ummtu+k\ncUs9LHgR4pOcm7be+0/obIfCS4aiO6YXUR38qkpHpxLnkuB7iRhzhvN6lb2VTXx05Dg7yxrIz0hi\n4aR0Zk1IIzHOedCJqlJa18JHR+p4d28VsyekERNz4r+HvIxRpI2KY/XmEibnJPPMiiWMSwse+gDs\nf4up7Tt5rXMRV5ZvgP/9Etz0OLzy97D9BZh9E5x1abi7bvoQ1cH/g5eLeWr9EUQgITaGuJgYm5gy\nEaXd46XNt3laUryL5nZnnj42Rhjle8KV16u4feWj4lx885Kzu32GiHD59DHsr2risa8s6raC5ySq\n8M6/05mSi1zxOOJ5F9bcBf85G9rdsOxHsHQl2EBrWEVt8B93t/PcphLOPyuLokkZtHm8tHd6h7tZ\nxgwqlwjTx6dyzqQMCrKSqGpqY8uROrYeraMl4G7cgqxkFk7MYPr4FOJcJ6/5eOBz87r9FdCrA2/D\n0Q24rvkFV86dBHwZPK3w3kNw0xMw9cpB7J0ZqOgL/tZ6aK7l+WIv7R4vP/j0TGaMTx3uVhkzJMak\nJHLlrHFcOWvcyQfb3VC1A5IyIXVCt0Mhhb4qrPs5pObCgi+fKF/8deefGTGiL/hfvAN2r+UmSWVu\n6gxmfPI+bLM/O02UUXVuomqudf7VHYaGgPsyU/MgfzGMSneOt9RC83Hf11rwtJyoG5cEWWdBynhn\nxc7VD0BsH9NBZthFX/DX7MOdMZ0/V41hedIR2PTEcLfImOGRkAqjMpwRfuFFTnhnTobGCij50HlQ\nSkczjMp06qTlwfi5zuu4JPwXxFrroXY/1OyD8fNg4d8Ma7dM/6Iv+BvL2TjqCn4cezOf/s4y8F3g\nMsYE+uZwN8CEUXQFf7sb2hrY2BzPjYvy/KsajDEmmkTXo28aywE41pnOF86dOMyNMcaY4RGVwZ8y\nJp+pY1OGuTHGGDM8oir4m6pLAJg6+ex+ahpjTOSKquA/eng/ALOnTxvmlhhjzPCJquCvLj9Mi8Yz\na3J+/5WNMSZCRVXwt9WW0hCXTVysreYxxkSvqAn+kuPNjO6oRkcHuVXdGGOiSNQE//v7ahjDcZKy\ngj0vxhhjokfUBP9f91czNqaOlBwLfmNMdIuKO3dVlS17S0im1dlIyhhjolhII34RWS4iu0Vkn4jc\nE+T4dBH5QETaROR7p3LuYHr2wyM8tf6w/9/u8kYA9lQ0Edvs3LxlwW+MiXb9jvhFxAU8DFwBlAAb\nRWSNqu4IqFYL3A1cP4BzB839f9jR7eESInDDglyyRycwVo47hSljw/GtjTHmjBHKVM9iYJ+qHgAQ\nkWeB6wB/eKtqJVApItec6rmD6d3vX4qiALR1eHlq/WGefP8Q7R4vX09rhjZsxG+MiXqhBH8ucDTg\nfQlwboifH/K5InI7cDvAxIkD20Ct57NA7716Bn9zfgGr1u3n2rbNsBNIseWcxpjoNmJW9ajqo6pa\npKpFOTk5g/a5uemj+Mn1s5mT1gLxoyHBNmczxkS3UIK/FAjc4yDPVxaK0zl3cDWW2WjfGGMILfg3\nAlNEpFBE4oFbgDUhfv7pnDu4Gitsft8YYwhhjl9VPSJyF/A64AKeUNXtInKH7/gqERkHbAJSAa+I\nrARmqmpDsHPD1Zk+NZZBXtGwfGtjjBlJQrqBS1XXAmt7lK0KeF2OM40T0rlDTtV5CItN9RhjzMi5\nuBtWrfXgabGpHmOMIVqC3/fIRUbbzVvGGBMlwV/mfLURvzHGRNgmbf/fFOhocV6LwOLb4fIfnBjx\n2xy/McZEWPDP/wJ0djiva/bCXx6AcXMCRvwW/MYYE1nBf8X9J1572uHJ5bDm2zDpfEhIg/jk4Wub\nMcaMEJE7xx8bDzc96Uz57HnNRvvGGOMTucEPkDEJrnvEeW3bMRtjDBBpUz3BzPg0XPMLSLVHLhpj\nDERD8AMsWjHcLTDGmBEjsqd6jDHGnMSC3xhjoowFvzHGRBkLfmOMiTIW/MYYE2Us+I0xJspY8Btj\nTJSx4DfGmCgjqjrcbTiJiFQBhwd4ejZQPYjNORNEY58hOvsdjX2G6Oz3qfZ5kqrmhFJxRAb/6RCR\nTaoaVU9Vj8Y+Q3T2Oxr7DNHZ73D22aZ6jDEmyljwG2NMlInE4H90uBswDKKxzxCd/Y7GPkN09jts\nfY64OX5jjDF9i8QRvzHGmD5ETPCLyHIR2S0i+0TknuFuT7iISL6IvC0iO0Rku4h8x1eeKSJ/FpG9\nvq8Zw93WwSYiLhHZIiKv+N5HQ5/TRWS1iOwSkZ0icl6k91tE/s73/+1iEfm9iCRGYp9F5AkRqRSR\n4oCyXvspIvf68m23iFx5Ot87IoJfRFzAw8BVwEzgVhGZObytChsP8F1VnQksAb7l6+s9wJuqOgV4\n0/c+0nwH2BnwPhr6/EvgNVWdDszD6X/E9ltEcoG7gSJVnQ24gFuIzD7/FljeoyxoP33/jd8CzPKd\n84gv9wYkIoIfWAzsU9UDqtoOPAtcN8xtCgtVLVPVj3yvG3GCIBenv//XV+3/AtcPTwvDQ0TygGuA\nxwKKI73PacBFwOMAqtquqnVEeL9xngw4SkRigSTgGBHYZ1V9F6jtUdxbP68DnlXVNlU9COzDyb0B\niZTgzwWOBrwv8ZVFNBEpABYAG4CxqlrmO1QORNrT5R8Cvg94A8oivc+FQBXwpG+K6zERSSaC+62q\npcADwBGgDKhX1T8RwX3uobd+DmrGRUrwRx0RGQ08D6xU1YbAY+os1YqY5Voi8mmgUlU391Yn0vrs\nEwssBH6tqgsANz2mOCKt37457etwfulNAJJF5EuBdSKtz70JZz8jJfhLgfyA93m+sogkInE4of+0\nqr7gK64QkfG+4+OByuFqXxgsBa4VkUM403iXichTRHafwRnVlajqBt/71Ti/CCK538uAg6papaod\nwAvA+UR2nwP11s9BzbhICf6NwBQRKRSReJyLIGuGuU1hISKCM+e7U1UfDDi0BviK7/VXgJeHum3h\noqr3qmqeqhbg/G/7lqp+iQjuM4CqlgNHRWSar+hyYAeR3e8jwBIRSfL9f/1ynOtYkdznQL31cw1w\ni4gkiEghMAX4cMDfRVUj4h9wNbAH2A/883C3J4z9vADnz79PgI99/64GsnBWAewF3gAyh7utYer/\nJcArvtcR32dgPrDJ97/3S0BGpPcbuB/YBRQDvwMSIrHPwO9xrmN04Px197W++gn8sy/fdgNXnc73\ntjt3jTEmykTKVI8xxpgQWfAbY0yUseA3xpgoY8FvjDFRxoLfGGOijAW/McZEGQt+Y4yJMhb8xhgT\nZf5/FsYaqkwpuc8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x24398ac13c8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['training', 'validation'], loc = 'upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training and validation accuracy seems to improve after around 60 epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 9376/10000 [===========================>..] - ETA: 0s"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy:  0.2144\n"
     ]
    }
   ],
   "source": [
    "print('Test accuracy: ', results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Weight Initialization\n",
    "- Changing weight initialization scheme can significantly improve training of the model by preventing vanishing gradient problem up to some degree\n",
    "- He normal or Xavier normal initialization schemes are SOTA at the moment\n",
    "- Doc: https://keras.io/initializers/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# from now on, create a function to generate (return) models\n",
    "def mlp_model():\n",
    "    model = Sequential()\n",
    "    \n",
    "    model.add(Dense(50, input_shape = (784, ), kernel_initializer='he_normal'))     # use he_normal initializer\n",
    "    model.add(Activation('sigmoid'))    \n",
    "    model.add(Dense(50, kernel_initializer='he_normal'))                            # use he_normal initializer\n",
    "    model.add(Activation('sigmoid'))    \n",
    "    model.add(Dense(50, kernel_initializer='he_normal'))                            # use he_normal initializer\n",
    "    model.add(Activation('sigmoid'))    \n",
    "    model.add(Dense(50, kernel_initializer='he_normal'))                            # use he_normal initializer\n",
    "    model.add(Activation('sigmoid'))    \n",
    "    model.add(Dense(10, kernel_initializer='he_normal'))                            # use he_normal initializer\n",
    "    model.add(Activation('softmax'))\n",
    "    \n",
    "    sgd = optimizers.SGD(lr = 0.001)\n",
    "    model.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = ['accuracy'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model = mlp_model()\n",
    "history = model.fit(X_train, y_train, validation_split = 0.3, epochs = 100, verbose = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XeYVdW5+PHvO2fKmc70PszQGToMiAoKVtQoFqIYzY25\nMUQTNd57kxuS3MT0m+R6vaZYgiXlZ5QoirEgogSDWJBeBhgYZobpvfdy1u+PfZjGAAeYes77eR6e\nc/baa52zFuI7a9699tpijEEppZTn8BruDiillBpaGviVUsrDaOBXSikPo4FfKaU8jAZ+pZTyMBr4\nlVLKw7gU+EVkmYhkikiWiKw+Q735ItIhIit6lOWKyAER2SsiOwei00oppc6f99kqiIgNeAK4GigA\ndojIG8aYQ/3U+xWwqZ+PWWqMqRiA/iqllLpArsz4FwBZxphsY0wbsBZY3k+9B4FXgbIB7J9SSqkB\ndtYZP5AA5Pc4LgAu6llBRBKAW4ClwPw+7Q3wvoh0An8wxqw52xdGRkaalJQUF7qmlFIKYNeuXRXG\nmChX6roS+F3xOPAdY4xDRPqeW2SMKRSRaOA9ETlijNnat5KIrAJWASQnJ7Nzp14OUEopV4nICVfr\nupLqKQSSehwnOst6SgfWikgusAJ4UkRuBjDGFDpfy4D1WKmjUxhj1hhj0o0x6VFRLv3QUkopdR5c\nCfw7gIkikioivsBK4I2eFYwxqcaYFGNMCrAO+Lox5nURCRSRYAARCQSuAQ4O6AiUUkqdk7Omeowx\nHSLyAPAuYAOeN8ZkiMh9zvNPn6F5DLDemf7xBl40xmy88G4rpZQ6XzISt2VOT083fXP87e3tFBQU\n0NLSMky9ci92u53ExER8fHyGuytKqQEgIruMMemu1B2oi7uDrqCggODgYFJSUujnArI6B8YYKisr\nKSgoIDU1dbi7o5QaYqNmy4aWlhYiIiI06A8AESEiIkJ/e1LKQ42awA9o0B9A+neplOcaVYFfKaVG\nuz151ew6UTWsfdDA76KamhqefPLJc253/fXXU1NTc8Y6P/zhD3n//ffPt2tKqVHk++sP8tBLexnO\nhTUa+F10usDf0dFxxnYbNmxgzJgxZ6zzk5/8hKuuuuqC+qeUGvma2zrJLK2nsKaZ/QW1w9YPDfwu\nWr16NcePH2f27NnMnz+fxYsXc9NNN5GWlgbAzTffzLx585g2bRpr1nRvR5SSkkJFRQW5ublMnTqV\nr371q0ybNo1rrrmG5uZmAO655x7WrVvXVf+RRx5h7ty5zJgxgyNHjgBQXl7O1VdfzbRp07j33nsZ\nO3YsFRW64alSo8mh4jo6HdZMf8PB4mHrx6hZztnTj9/M4FBR3YB+Zlp8CI/cOO2053/5y19y8OBB\n9u7dywcffMANN9zAwYMHu5ZDPv/884SHh9Pc3Mz8+fO57bbbiIiI6PUZx44d46WXXuKZZ57h9ttv\n59VXX+Xuu+8+5bsiIyPZvXs3Tz75JI8++ijPPvssP/7xj7niiiv47ne/y8aNG3nuuecGdPxKqcF3\noMBK+06NC+GdAyWsXjZlWBZa6Iz/PC1YsKDXGvjf/va3zJo1i4ULF5Kfn8+xY8dOaZOamsrs2bMB\nmDdvHrm5uf1+9q233npKnW3btrFy5UoAli1bRlhY2ACORik1FPYX1BId7Mc9l4wlr6qJjAGewLpq\nVM74zzQzHyqBgYFd7z/44APef/99PvnkEwICAliyZEm/a+T9/Py63ttstq5Uz+nq2Wy2s15DUEqN\nAp8+DSFx7C8cw8zEUK5Oi+V76w/yzsFipieEDnl3dMbvouDgYOrr6/s9V1tbS1hYGAEBARw5coRP\nP/10wL//0ksv5eWXXwZg06ZNVFdXD/h3KKUGQfYHsPE7ODZ+j+zyOmYkjCE80JeLx0Ww4UDJsKzu\n0cDvooiICC699FKmT5/Ot7/97V7nli1bRkdHB1OnTmX16tUsXLhwwL//kUceYdOmTUyfPp1XXnmF\n2NhYgoODB/x7lFIDqLUB3ngQvO141RUwl6PMTLJm+NfNiCWnopHM0v4nlINp1GzSdvjwYaZOnTpM\nPRp+ra2t2Gw2vL29+eSTT7j//vvZu3fvBX2mp/+dKjXo3v4W7HgW7l5Hx4t3sbbtUq5b/RIRQX5U\nNLSy4Ofv88AVE/n3qydd8Fe55SZtni4vL4/bb78dh8OBr68vzzzzzHB3SSl1Jjkfwo5nYOHXYcJV\n7A28hBsd2wm1W6t4IoP8WDgugj9/lMOcpDEsnRI9ZF3TwD9KTJw4kT179gx3N5RSrijYCa99FcJS\n4YofAPBK28X8in9A9haYdC04HKwJeIpM7xPc9qdvce+icfznsin4eg9+Bl5z/EopNVCMgc+egeeX\ngc0H7ngBfAOoaWrjtdrJtHiHwn5rkQYf/5agY68zr2MP35vZyLPbcrjtqY9pbB38lXwa+JVSygXt\nnY6zV3rzIdjwLZhwJXxtK8ROB6z1++14U516A2RugGPvw+afwOTrwTeIVf4f8IcvzmNBajiBfoOf\niNHAr5RSZ2CM4YktWaT9cCM7cs+wq2bFMdj9F1jwNVj5Evh332R5oNDalyd4/p3Q3gQv3QFhY+GW\nP8DM2yHjNa5N9eUHn0sb7OEAGviVUuq0Glo7uP+F3fzPu5m0dxr+mVl++soZ6wGBRQ+DV+/Qui+/\nhtTIQIImLILQJBAb3P4XsIdA+r9CRwvse2lwB9ODBv5BEhQUBEBRURErVqzot86SJUvou2y1r8cf\nf5ympqauY1e2eVZKXbiapjZueeIjNh0q4b9umMr0hBB2551642RLe6d1E1bGeki+GELie50/WFjL\nlswyLh4fYf1AuP3P8KU3IXaGVSF2BiQugJ3PW9cIhoAG/kEWHx/ftfPm+egb+F3Z5lkpdeE2ZZRy\nrKyBNV9M597F45ibHMa+/Jqu3TXB+o3gol9sZt3G96HsEEy7pddnNLd18tDaPUQE+vGf1062ChPm\nQfJFvb9s/legMgty/jnYwwJcDPwiskxEMkUkS0RWn6HefBHpEJEV59p2pFu9ejVPPPFE1/GPfvQj\nfvazn3HllVd2baH897///ZR2ubm5TJ9uXeBpbm5m5cqVTJ06lVtuuaXXXj33338/6enpTJs2jUce\neQSwNn4rKipi6dKlLF26FOje5hngscceY/r06UyfPp3HH3+86/tOt/2zUsp1O3KrCAvw4cqp1vr6\nOcljaGzr5FhZfdfM/OOsCmqb26nc/jcMAmnLe33Gz94+RE5FI4/dPosxAb6n/7K0m61rAjuGZtfd\ns14+FhEb8ARwNVAA7BCRN4wxh/qp9ytg07m2PWfvrIaSAxf0EaeInQHX/fK0p++44w4efvhhvvGN\nbwDw8ssv8+677/LQQw8REhJCRUUFCxcu5KabbjrtNqtPPfUUAQEBHD58mP379zN37tyucz//+c8J\nDw+ns7OTK6+8kv379/PQQw/x2GOPsWXLFiIjI3t91q5du/jjH//I9u3bMcZw0UUXcfnllxMWFuby\n9s9KqdPbeaKa9JTwrv+f5yRZF2szsnKY8rcvwcL7+WfRJfjahKscH1EYNpfE4Jiu9u8dKuWv2/P4\n2mXjuGRCZL/f0cXHDrPvgsNvQkcrePuduf4FcmXGvwDIMsZkG2PagLXA8n7qPQi8CpSdR9sRb86c\nOZSVlVFUVMS+ffsICwsjNjaW733ve8ycOZOrrrqKwsJCSktLT/sZW7du7QrAM2fOZObMmV3nXn75\nZebOncucOXPIyMjg0KEz/2zctm0bt9xyC4GBgQQFBXHrrbfy4YcfAq5v/6yU6l95fSs5FY3MT+le\nmTM2IoDwQF/aDm+C6hx45z+xHVrPnSkNTPAq4k91c6lvaQfgUFEd//HyXqYnhPAf10x27UuXrIYH\ndw960AfX7txNAPJ7HBcAvRJUIpIA3AIsBeafS9sen7EKWAWQnJx85h6dYWY+mD7/+c+zbt06SkpK\nuOOOO/jrX/9KeXk5u3btwsfHh5SUlH63Yz6bnJwcHn30UXbs2EFYWBj33HPPeX3OSa5u/6yU6t9O\n57LN9JTwrjIRYU7SGCILt0FABM2hE/h+0W+oaL0II16sb55L2CcnuHZaLF98bjuBft48ddc81+/E\n9Ru6TRcH6uLu48B3jDEu3OHQP2PMGmNMujEmPSoqaoC6NbDuuOMO1q5dy7p16/j85z9PbW0t0dHR\n+Pj4sGXLFk6cOHHG9pdddhkvvvgiAAcPHmT//v0A1NXVERgYSGhoKKWlpbzzzjtdbU63HfTixYt5\n/fXXaWpqorGxkfXr17N48eIBHK1SnmtHbjV2Hy+mx/feK39OUghz2nbTlrKE1yb/mjwTQ0LFR0jq\nZcyeMpE1W7O5+9ntiAh/vfciksIDhmkEZ+bKjL8QSOpxnOgs6ykdWOvMhUUC14tIh4ttR41p06ZR\nX19PQkICcXFx3HXXXdx4443MmDGD9PR0pkyZcsb2999/P1/+8peZOnUqU6dOZd68eQDMmjWLOXPm\nMGXKFJKSkrj00ku72qxatYply5YRHx/Pli1busrnzp3LPffcw4IFCwC49957mTNnjqZ1lDofFVmQ\n8wHkbIXaQlrqb2N20pxTZuuLgoqJlDqOhCzkvZw23gr6ES+F/QEWfoNvBkzkpt9/hAisXbWQcVFB\nwzMWF5x1W2YR8QaOAldiBe0dwBeMMRmnqf8n4C1jzLpzbXuSbss8NPTvVCng4Guw7svW+5AEHOKF\nqSngo5RvcNk9P4MeizVat/wPfv/8Gb+bs4Endtaxcn4yP7qp+4mAb+wrYkpsMJNihv5ZGQO6LbMx\npkNEHgDeBWzA88aYDBG5z3n+6XNt60rHlFJqIDW1dfDPzHIyiurIKKrFz9vGU3fPRQp2gE8A3LcN\nwsex/fAJKl9axedO/B7+lgu3PQs+/gD45X7Aca9UntnTQEu7g8sn9U5L3zQr/tQvHoFc2g3IGLMB\n2NCnrN+Ab4y552xtlVJqqP16YyZ/+jgXm5cQFuBDRUMb5Q2tRJcfgchJEDEegE8L2/hdx0NcdU0O\n9s3/Be//2FpQ0loP+Z9SEHE7dfkd+Hp7cdG48LN868g0qu7cHYlPCxut9O9SeZoPj5Vz6YQIMn58\nLb9eYS2lLqhuhvJMiOq+PrfzRBVT40KxL34Q5n8Vtj8FudusB6s4OjDjrgRgQUo4Ab6j85Emoybw\n2+12KisrNWANAGMMlZWV2O324e6KUkOitK6F4+WNXD4pCruPjcQwa7VNSVk51BVClLXWvr3Twe4T\nNcw/uYzz6h9bD1N5/etw+A3wCSBpjnUX/ZLJI3P1oStGzY+rxMRECgoKKC8/w+54ymV2u53ExMTh\n7oZSQ+LT7EoALh5n3UGbMMbK2TcVOS85Omf8h4rqaG7v7A78voFwy9PWg1VqTsDEaxkfG8HaVQuZ\nkzx698waNYHfx8eH1NTU4e6GUmoU+uR4JSF2b9LiQwAI9PMmItAXypxbvzhn/AeLrH3zZyX1WL+f\nvBAu/gZ88nvrASvAwnERQ9f5QTBqAr9SSp2vj49XctG4CGxe3UszE8P88a89BjY/CEsBILu8EbuP\nF/Gh/r0/4IofQHAszLpzCHs9eEZNjl8ppc5ZbSFlRz4mr6qJi/vM0hPDAohoyoHIieBlAyC7vIHU\nyCC8vPpstOhjh0setB6c4gY08Cul3FNbE/zlJqLXXsdq7xe5JLV30E4M8yehIx8T2b2JWnZFI+Oi\nAoe6p0NOA79Syj2990OozOJA8CLu836LyRtuh+ru/bRSQiCBchpDJwDQ2tFJflUT4yM18Cul1OiT\ntRl2PINZ+HW+1vbvPBP7Q6TiqLU6p6MNgAm2ErzEUOo3FoC8yiYchhG9x85A0cCvlBp16lvaueXJ\nj/jd5mOnnmyqgr9/A6KmkDf7WxTVtmCfvQJuew7qiyDzbQCSOqzZf66XtY/k8fJGAE31KKXUsCvc\nbW2X4NTR6eCBF/ewJ6+G7TlVp9b/x0+hsRxuXcPHeVYwv3hchLUUMzQJdv0ZgIjmXNqNjaNt1o1Y\n2RUNAKRqqkcppYZRUxU8dzWsvQscnRhj+NGbGfzzaDmRQX4U1fTzkKHjW2DydRA3ix25VUQG+TE+\nKtBauTPnbsjeAtW5+FQeJU/iOFFjPTUru7yR6GA/gu0+QzzIoaeBXyk1cuX8Exwd1uu2x/jjR7m8\n8GkeX7t8HDfPjqeotrn3Ni7NNdZjEePnAJBV1sDUuODu52DPuRsQ2PMClB+hxDfF2q8HaymnJ6R5\nQAO/UmokO74F/EJg2q2YLb9gy6bXuXxSFN+5dgrxY/xpaXdQ3dTeXb/EeqodcbMwxpBd3tg7dROa\nCBOugt3/D6pzqAtKpaC6CTi5lNP9L+yCBn6l1EhljJWWSb0MbvwNnaFj+bX8ls9N8MXLS4h37rfT\nK91TvM96jZtNeX0rDa0djOubs5/3JWgoAeOgLXwShTXNVDS0UtPUfmpdN6WBXyk1MlVlQ00ejFsC\n9hAyLn6cSGq5uPQlAOLHWLvLFvYM/EV7ISQRAiN7rNLpM4uftAwCowGwxUylvdPwyXFrE7fxOuNX\nSqlhlO18xvT4KwA4SCp5JprwtiKArhl/cd8Zf9wsq7lzlc4peXubD6R/GfxCCEm0Hj269Wh5/3Xd\nlAZ+pdTIdHwLhCZD+DgAcisaqSAM/xYrSEcE+uLr7UVRbYtVv7UeKrMgfjZwhg3XAC7/Djywk4RI\na2vlrcfK8bV5de3T7+408CulRp7ODsjZCuOXdD3sPKeikSa/SKShBAARIWGMf3eqp+QAYLpn/OUN\npEQEnrrhGlhLO4NjuvblL61rZWxEQK/dO92ZBn6l1MhTtBta62Dc0q6inIpGOgKioaHUuvALxIXa\nu1M9XRd2Z3XVP1vO3u5jIyrYD/CcNA+4GPhFZJmIZIpIlois7uf8chHZLyJ7RWSniCzqcS5XRA6c\nPDeQnVdKuanjWwCxLuwCnQ5DXlUTttA4aG/qupM3fow/RTXOVE/RXgiKheBY2joc5Fc3uxTME8Os\nWb+nLOUEFwK/iNiAJ4DrgDTgThFJ61NtMzDLGDMb+Ffg2T7nlxpjZhtj0gegz0opd5e9xcrVB1iP\nQCysbqa90xAQHm+dbygFrMBfWt9Ce6ej14XdvKpGOh3GpcCf5Mzre8pSTnBtxr8AyDLGZBtj2oC1\nwPKeFYwxDab79rlAQJ+IrpQ6Py11ULCjd5qn0lqaGRqdbBXUFwMQH2rHGCirrIKKzK7A37WUM/Ls\ns3id8fcvAcjvcVzgLOtFRG4RkSPA21iz/pMM8L6I7BKRVRfSWaWUBzi+2dqmYeI1XUW5FVYgj463\ntlCmvnvGD1CbsxeMo9eKHnAtbz83OYzwQF8mxXhO4B+wZ+4aY9YD60XkMuCnwFXOU4uMMYUiEg28\nJyJHjDFb+7Z3/lBYBZCcnDxQ3VJKjTaZ74B/OCQt6CrKqWgk0NdGeIy1hTLOlT0nA397wR6rvMeK\nnigXN1y7Ki2G3WlXD+AARj5XZvyFQFKP40RnWb+cQX2ciEQ6jwudr2XAeqzUUX/t1hhj0o0x6VFR\nUS52XynlVjo74Oi71t21zufgghX4U6MCEXsoePtD/cnAb92961u+HwIiIMRKRmRXNHpUzv5cuRL4\ndwATRSRVRHyBlcAbPSuIyARxbn8nInMBP6BSRAJFJNhZHghcAxwcyAEopdxI/qfQUmNtq9xDbmUj\nKRGB1pr+4JiuwB/g682YAB9Cao9A7MyuNf/WTpuek7o5V2dN9RhjOkTkAeBdwAY8b4zJEJH7nOef\nBm4D/kVE2oFm4A5jjBGRGKz0z8nvetEYs3GQxqKUGu2ObACbb9c2DYC1NLOqiZtmOVf0BMV2reoB\niA+xE1GbB1FWm+rGNqqb2q09+FW/XMrxG2M2ABv6lD3d4/2vgF/10y4bmHWBfVRKeQJjIHMDpF4O\nft2z9fxq61m4XdsrB8dA6aGu82nBjdhrWiDCemj6affoUV30zl2l1MhQnmk9RKVvmse5oielK/DH\n9ZrxT/Mrs944A/+5LOX0VBr4lVIjQ6YzqTBpWa/iHGfgT41wBv6gGGs7hzarfLyXle9vCE4FrKWc\nPjbpWp+vTqWBXyk1MmS+A3GzIbT3bUI5FY2MCfAhLNDXKgiOtV6dF3gTOgtoMn4UOcIwxrA7r5qx\nEYF42zS8nY7+zSilhl9DuXW37uTrTznVtaLnpKAYZxsr3RPRmk+uiaWotpVXdxfyWU4Vdy7Qe4HO\nZMBu4FJKqfN29B3AnJLfB8gpb2ThuIjugj4z/qD6HLJNHJknqvnTR7ksSA3ny5ekDH6fRzGd8Sul\nht+RDdZDV2Jn9Cr+ILOMotoWJscGdxcGOQN/Qyl0tGKryyOHeJ7YkkWnMTy6Ylb/e/CrLhr4lVLD\nq63J2o1z8nVdN2AB5FU28c21e5kaF8K/XJzSXT8gHLx8rBl/dS5iHFT5JeEw8L3rp5Ic4RlP0boQ\nmupRSg2v7C3Q0dIrzdPc1snXXtiFMYY/3D0Pf9/u7Rusu3djrcBfcQwA//gpXGWL4a6LNLfvCg38\nSqnhdWQD+IVCStfzm/j++gMcKanj+Xvm9z+DD4qxNmqrzALg23deD/ZQRDTF4wpN9Silho+jE45u\nhIlXg83aSXNPXjWv7SnkwSsmsnRydP/tgmOtrZkrj0FgNOI/RoP+OdDAr5QaPvmfQVMFTOlexvnC\np3kE+tpYddm407c7OeOvyILIiUPQUfeigV8pNXwy37Yu1E6wHt9R09TGW/uLuHlOAkF+Z8hEB8dC\nczWUHe7aqkG5TgO/UmpYGGNoOfgWjpRFYA8FYN2uAlo7HNy9cOyZG5+8iau1VgP/edDAr5QaFh9s\nfAV7XQ6vN83CGIPDYfjr9jzmjQ1jalzImRsHx3W/11TPOdNVPUqpIdd2dDMXb/8G2STwo9w0sjcd\nZeG4CHIqGnnoShdm8MEx3e8jNPCfKw38SqmhlbkRr799keOOWOpvf4Ubjrbx+y1ZrN9TSFiAD9dN\njzv7Z5y8e9fLG8LOkhZSp9DAr5QaOsf/gfnb3Rx1JPF44q9YM30Kc6c6KK9v5f3DZXztsnHYfWxn\n/5zASBAvCEvpWgaqXKeBXyk1NCqPwyv3UGkfyx1Vq/nLsvkAeNu8+N2dc/nr9hOsmJfo2md52awL\nvJrmOS8a+JVSg6+lFl5aiUNs3N34MBenpTInOazrtL+vjXsXn2Hdfn8+93+9L/Iql2ngV0oNLkcn\nvPpVqMrm44uf5chmX365dACWYPazhbNyjUvLOUVkmYhkikiWiKzu5/xyEdkvIntFZKeILHK1rVLK\nzR18DY69C8t+yXaThs1LmBoXfPZ2atCcNfCLiA14ArgOSAPuFJG0PtU2A7OMMbOBfwWePYe2Sil3\nduIjaxO29K9wvLyB5PAA/LxduICrBo0rM/4FQJYxJtsY0wasBZb3rGCMaTDGGOdhIGBcbauUcnMF\nOyFxHnh5cbyskfFRgWdvowaVK4E/AcjvcVzgLOtFRG4RkSPA21izfpfbKqXcVGsDlGVA4nw6HYac\nikbGRwUNd6883oBt2WCMWW+MmQLcDPz0XNuLyCrn9YGd5eXlA9UtpdRwKt4LxgEJ6RRUN9HW6WB8\ntAb+4eZK4C8EknocJzrL+mWM2QqME5HIc2lrjFljjEk3xqRHRUW50C2l1EjV3ukgr7IJCnZYBYnp\nZJU1AOiMfwRwJfDvACaKSKqI+AIrgTd6VhCRCeJ8CoKIzAX8gEpX2iql3M+ardlc9X//pC33Mwgf\nDwHhHC8/Gfg1xz/czrqO3xjTISIPAO8CNuB5Y0yGiNznPP80cBvwLyLSDjQDdzgv9vbbdpDGopQa\nId7eX0xbRyeOgh0w6QoAjpc1Ehnky5gA32HunXLpBi5jzAZgQ5+yp3u8/xXwK1fbKqXcV35VE4eK\n64inEntLOSRaWzMcL29gnKZ5RgTdj18pNaA2HSoF4NpQ54K+xHTACvya3x8ZdMsGpdSA2pRRwuSY\nYG4KKqKlwIeW4Ek4GtuobmrX/P4IoTN+pdSAqWpsY0duFddMi2Fi+xEOmFS259V3r+jRpZwjggZ+\npdSAef9wKQ4Dy6aEE1h5kANMZHt2VdeKngma6hkRNNWjlBowmzJKSRjjT5otH+lspT5yNp9mV+Il\n4OftRcIY/+HuokJn/EqpAdLU1sGHx8q5Oi0GOfI2AMHjL+ZwSR2786oZFxWEl5cMcy8VaOBXSl2I\n/a/AK/dASx1bj5bT2uFgRWQubHsMpq9geto0jIHdeTV6YXcE0VSPUur8HXkLDr0OVTkciv4FCT51\nTPv4Eetu3RsfZ5YtAD9vL1o7HLqUcwTRwK+UOn8NpRAcD+VH+ELlfVzhE4y01MEX14NfMH7A3OQw\nPsmu1BU9I4imepRS56++BMZeAne/Rmh7BbMdGXDD/0LMtK4qC8dFADAuUlM9I4XO+JVSZ7VuVwHG\nGD6f3mOzXWOcM/5YSLmUbwb+ioUBhfzrnLt6tb3zoiS8BNLiQoa41+p0NPArpc7q+W05AL0Df2s9\ntDdBUAwAO5vjiJ4w55S20cF2Hrxy4pD0U7lGA79S6qxK61q6nqfapcHak4fgWNo7HVQ1thEZ5DfU\nXVPnQQO/UuqM2jocVDa2AdDS3ondx/mg9PoS6zUohirn+ahgDfyjgV7cVUqdUVl9S9f70rru9z1n\n/OX1rQA64x8lNPArpc6oZ7Avru0R+HvM+MsbNPCPJhr4lVJnVFLb2uN9zxl/CXjbwR5KhXPGH62p\nnlFBA79S6oxOP+MvhaBoENEZ/yijF3eVUmdUWteCr80LPx8vSmqbu080lEBQLAAV9W0E+Xnj72sb\npl6qc6GBXyl1RiV1LcSE+hHg433qjD9qEgDlDa1EBulD1EcLTfUopc6otK6FmGA7saF2Sur65Pi7\nZvytupRzFHEp8IvIMhHJFJEsEVndz/m7RGS/iBwQkY9FZFaPc7nO8r0isnMgO6+UGnylda3EhNqJ\nC7V3z/jbm6GlFoKtu3atGb8G/tHirIFfRGzAE8B1QBpwp4ik9amWA1xujJkB/BRY0+f8UmPMbGNM\n+gD0WSnQYOndAAAY+klEQVQ1RIwxlNS2EBtizfgrGlpp63B0r+E/OePXwD+quDLjXwBkGWOyjTFt\nwFpgec8KxpiPjTHVzsNPgcSB7aZSajjUtXTQ3N5JbIg14zfGeUNXfffNW20dDmqa2jXVM4q4EvgT\ngPwexwXOstP5CvBOj2MDvC8iu0Rk1ekaicgqEdkpIjvLy8td6JZSarCVOXP60SF+xIZaz8strm2x\n8vsAQTFUNupSztFmQFf1iMhSrMC/qEfxImNMoYhEA++JyBFjzNa+bY0xa3CmiNLT00/ZD0opNfRO\nXsyNDbETFmit2imubYGW7hl/RY3u0zPauDLjLwR67MVKorOsFxGZCTwLLDfGVJ4sN8YUOl/LgPVY\nqSOl1Chw8k7d2FArx2+VNVszfrFBQCTlDVYdXc45ergS+HcAE0UkVUR8gZXAGz0riEgy8BrwRWPM\n0R7lgSISfPI9cA1wcKA6r5QaXGXOrRhiQuyE2H0I8nOu5T95166XFxX1OuMfbc6a6jHGdIjIA8C7\ngA143hiTISL3Oc8/DfwQiACeFBGADucKnhhgvbPMG3jRGLNxUEailBpwJbUthPr7dG3FHBtqd/4W\nUNL1ABbdrmH0cSnHb4zZAGzoU/Z0j/f3Avf20y4bmNW3XCk1OpTUWUs5T+pay+9VCqHWGo/y+laC\n7d7d+/SrEU/v3FVK9e/Nh7mk5K/EhHYH/tgQ54y/ofeMP0pn+6OKBn6l1KlqC2HXn1jS9C4xPXL3\ncaF2KusbMY0V1kPWsbZr0DTP6KKBXyl1qozXAEMqhaQEdO/HHxvqT5ipQzBdM/6KBt2nZ7TRwK+U\nOtWBdTh8AgGY6uhaqEdcqJ1ocd6kfzLVU687c442GviVUr1VZEHxXkpn3EenEVKaMrpOxYbaiZYa\n6yA4ltaOTupaOnTGP8po4FdK9XZwHSBkxi/nsBlLdO2+rlNxPQN/UAwVDdYafs3xjy4a+JXycO8d\nKqWwxvlkLWPgwDpIWUReeyi7HRMJKN8Ljk4AQv19iLfVWnWDYrqetasz/tFFA79SnsYY2PxTKM+k\nrcPBfS/sYs0/j1vnivdB5TGYsYKS2hb2MhGv9kYoOwyAiDDWt54GrxDw9qW8Xm/eGo008Cvlaaqy\n4cNHYd9aimub6XQYjpY2WOcOrgMvH5h6E6V1reQFTLfK87dbr8YwQ7IoIoqqxjYqGnTGPxpp4FfK\n05Q6L9ZWZVNYbaV4jpXVW2WH3oDxV0BAOKV1LXSEjIXAKCjYYZ0/9h7j2rP4Y9sVLH30A17bbe3X\nGKGrekYVDfxKeZquwH+cAmduv6KhjarqKqg5AUnWBroF1U3Eh/lD4gLI/8xKEX3w3zAmmXvu/y5p\ncSF8lltFiN0bP2/drmE0GdD9+JVSo0Cpc4PcymwKq5q6iguz9hMOEDmRupZ2ciubWDEvEXznQ+bb\nsO8lKNoNN/6WyQkRvPjVcN7NKKG9Ux+fMdpo4FfK05QeBATaG6mrKMTX24u2Dge1+Yes85GTyCis\nA2B6QijYL7LKN3wbxiTD7C8A1oXeZdPjhmEA6kJpqkcpT9JaD9W5kLwQAFN5nGnxIQT62ugoywTx\ngvBxHCi01urPSAiF+Dng5Q1tDXDZt8HmM4wDUANBA79SnqTUOaufehMA9vpcEsMCmBATjF9tNowZ\nC95+HCisI2GMPxFBfuDjbwX/MWNh1p3D2Hk1UDTVo5QnOZnfn7wM894PCG3OhzH+2L29iCg/AYlT\nADhYWGvN9k9a8bx1cVdn+25BZ/xKeZLSDPALhbBUOkOSSaaYxDB/JkYHkGyKaAkdR11LOzkVjcxI\n7BH4xyRD2Njh67caUDrjV8qTlGZAzDQQoTFoLKlV2QSE+WNvqMQu7eT4JFFcaG3JML3njF+5FZ3x\nK+UpHI7uwA9U+CYyVkpIDLUzwasIgGOdsRwosAL/DA38bktn/Ep5ito8aKvvCvyFXnGMl1YSfOvw\nb8kDYF9zNCdqakkY4094oN6N665cmvGLyDIRyRSRLBFZ3c/5u0Rkv4gcEJGPRWSWq22VUoOkvgSe\nvBhyt1nHJ+/YjZ0BwPFO60EqAXUnkMpj1EsQeyq9OVBYy8xEne27s7MGfhGxAU8A1wFpwJ0iktan\nWg5wuTFmBvBTYM05tFVKDYbcbVB2CF69FxorocR541aUtXLnUGuUVa8qGyqOUWkfy4GiOk5UNml+\n3825MuNfAGQZY7KNMW3AWmB5zwrGmI+NMc7nsfEpkOhqW6XUICneZ9141VQJbzwApQcgPBX8ggDY\nXx9MB95QdRwqjtEaOo76lg4AnfG7OVcCfwKQ3+O4wFl2Ol8B3jnPtkqpgVK8D2Kmw1U/hswNcGRD\nV37fGEN+bRs1fvFQtBcaSvCOntzVdHq8Bn53NqCrekRkKVbg/855tF0lIjtFZGd5eflAdkspz2OM\nFfjjZsFF98GEq8B0QoyV369paqeprZOm4LFw4iMAQpOsLGximD9hemHXrbkS+AuBpB7Hic6yXkRk\nJvAssNwYU3kubQGMMWuMMenGmPSoqChX+q6UOp3afGipsQK/lxfc/BRMuBqmXA/Q9ahFx5hx0Gk9\nNzc8eRqBvjZN83gAV5Zz7gAmikgqVtBeCXyhZwURSQZeA75ojDl6Lm2VUoOg2PmA9DhrgV1nQBS/\nifkFd9iTSAAKnA9g8Y2eAFmA2PCKGMfvvxBOUnjA8PRZDZmzBn5jTIeIPAC8C9iA540xGSJyn/P8\n08APgQjgSREB6HDO3vttO0hjUUqdVLwPxNaV088sqee3m4+RUVjLc/fM75rxB8c78/rhqeDty9Ip\n0cPVYzWEXLqByxizAdjQp+zpHu/vBe51ta1SapAV74OoydbOmkBeVSMAm4+Use1YBYXVzQT42giK\nn2TVj5g4XD1Vw0C3bFDKHRXv70rzAORWWk/aig2x87O3D5FX1UTCGH9kTLK1aVvczOHqqRoGGviV\ncjf1JdBQArHdwfxEZRPhgb781+emcqSkni2ZZSSE+YOXDe77EC59eBg7rIaaBn6l3E3xfuu1x4z/\nRGUjyeEB3DAjjrnJY+h0GBLGWGkgwsaCr17Q9SQa+JVyNyXOFT3OPXnAmvGnRAQgIvzgc9Z6/dTI\nwOHonRoBdHdOpdxN8T4IHw/2EABaOzopqm0mOcLaSWVOchhvPrCI8dEa+D2VBn6l3E3xPkiY13VY\nUN2MMZAS0Z3OmaE3aXk0TfUo5U6aqqAm75T8PsDYCM3jK4sGfqXcSdEe67VX4LeWco6N0NSOsmjg\nV8qdHNsE3nZIWthVdKKyiUBfGxG68Zpy0sCvlLswBjLfgXFLei3PPFHZyNiIQJzbqSilgV8pt1F+\nBGpOwKRlvYpPVDVpfl/1ooFfKXeR6dwSq0fg73QY8quaNL+vetHAr5S7yNwIcbMhJK6rqLi2mfZO\nozN+1YsGfqXcQUM5FOyAydf3Ku5a0aN77KseNPAr5Q6ObQIMTO6T3z8Z+HV7BtWDBn6l3EHmBghJ\n6LUjJ1grenxtXsSG2IepY2ok0sCv1GjX3gLHt8Cka+lwGN7cV0RLeydgzfiTwv2xeelSTtVNA79S\no92Jj6C9ESZfz5v7i3jwpT3c++edNLd1kutcw69UTxr4lRrtCnYCAskXs/VoBXYfLz46XsG9f9lB\nnq7hV/3QwK/UKJRVVs/P3z5ER6fD2o0zciIOn0A+PFbBNWmx/O/nZ/Hx8Uqa2jp1RY86hUuBX0SW\niUimiGSJyOp+zk8RkU9EpFVEvtXnXK6IHBCRvSKyc6A6rpSnMsbwnVcP8MyHOWzLqrACf9wsjpTU\nU9HQyuKJkdw6N5HHbp+Fr7cXMxLHDHeX1Qhz1v34RcQGPAFcDRQAO0TkDWPMoR7VqoCHgJtP8zFL\njTEVF9pZpRRsPFjCrhPVAGz6LIMldQUQN4sPj5UDsHhiFAC3zEnkczPj8bHpL/aqN1f+RSwAsowx\n2caYNmAtsLxnBWNMmTFmB9A+CH1USjm1dTj45cYjTIoJ4s4FSZQe3W6diJvFh8cqmBQTRGxo99JN\nDfqqP678q0gA8nscFzjLXGWA90Vkl4isOpfOKaV6+8snuZyobOJ7109lxbwkJjuyAWiOmM5nuVVd\ns32lzmQoHr24yBhTKCLRwHsicsQYs7VvJecPhVUAycnJQ9AtpUaXmqY2fvePLBZPjGTJ5GiMMdTb\n8ykljiMlnbR1OFg8MXK4u6lGAVdm/IVAUo/jRGeZS4wxhc7XMmA9Vuqov3prjDHpxpj0qCidtSjP\n1d7poK3D0avMGMMP/p5BfUs7379hKgAiwmyfE+xqS+LlHfn4entxUWrEcHRZjTKuBP4dwEQRSRUR\nX2Al8IYrHy4igSISfPI9cA1w8Hw7q5Qn+OHfM7j4vzezO6+6q+z3/8jizX1F/Mc1k5kSG2IVNlcz\npqWQg45U3j5QzIKUcPx9bcPUazWanDXVY4zpEJEHgHcBG/C8MSZDRO5znn9aRGKBnUAI4BCRh4E0\nIBJY73zyjzfwojFm4+AMRanRr7Wjkzf3FdHQ2sGdaz7lNyvnYIzhf987yq1zEvj6kvHdlUsOWG2i\nZkAJmuZRLnMpx2+M2QBs6FP2dI/3JVgpoL7qgFn9lCul+vHJ8UoaWjv4nxUzeemzPO7/6y58bF7M\nTR7DL26d0fvxicX7AJiRvgjeKmLJ5Ohh6rUabYbi4q5SykXvZpQS4Gvjxlnx3Dgrnv9ct5+DRbX8\n4Yvp2H36pHGK90FIAssvnc2MyRMZHxU0PJ1Wo44GfqVGCIfD8N6hUpZMjuoK8r+900r19PugdOcd\nuyKiQV+dE727Q6lh0tbhoNNhuo735FdT0dDKtdNie9XrN+i3NkDFMYjTTKo6dxr4lRpK7c1gDC3t\nnSx/4iNue+pjmts6obWBLfuz8bEJS6e4kKsvPQgYDfzqvGiqR6nBdvhNyFgPxfuhMguipvBq1Dc5\nXByCiOFvf3yML9U+xV3NNnJSniDE7nPmz+tsh49/BwjEzxmSISj3ooFfqcFUXwov/wsERkNiOqTd\nROuev3HXofuYEruMSK96xhZ/RJ59MtGObH7Y9EvoWALevv1/Xmc7vPoVOPIWXPNzCI7tv55SZ6CB\nX6nBdPBVMA740hsQNZmW9k5W7F3ICttavlT3Bth8eS36Qb6VdxE32j7hN9VPwMbvwOf+79TPOhn0\nD/3dCvqXPDD041FuwTMC/8FXwS8UJl413D1Rnmb/36w8fNRkAH67+RgHy9v51pd/iUQ/AjZfrveP\nZu1zn1FvvwUSfOCjxyFmGsy/t/dnbf6JBn01IDwj8G/6AXj7wYTd0N8KCaUGQ/lRKN4L1/4CsJ6a\ntWZrNivmJfa62coO/O1rC60VPjIXyg7Bxu9C8iUQk9b9WZ8+CXPu1qCvLpj7r+pproG6QqjKdq6E\nUGqIHHgZxAum34Yxhh+9cYgAXxvfvW7KKVVFBG+bF3jZYPmT4BcMr99npXeMgY2rwScArvzR0I9D\nuR33D/xlh7vfH/r78PVDeZSyumbM/lcg9XIIjuXdjBK2ZVXwH9dMJiLI78yNg6KsHH/xPtj2f5D5\nDhzfDEu+a51T6gJ5QOB3PiEyYgIccmlTUaUuyO68ah749RqkJpesuOtpbuvkp28dZkpsMHdd5OKz\nJtKWw/QV8M9fwdv/DlFTYMFXB7fjymN4QOA/DL7BcNF9UJEJZUeGu0dqFHM4jHXD1WmU1bdw/wu7\nuMPvY1rx5ebNYdzy5EcU1jTzo5umWekcV13/PxAQAfXFsOyXYDvL+n6lXOT+F3fLDkP0VJh6I2z4\ntpXuiT41x6pUX8YYMorqeP9wKZ8cr6SotpnS2lbaHQ7uuSSF7yyb0mvjtLYOBz/4yyZWtrzNzX7b\nMJNu4IuhM3hmazbLZ8ezcNw5PiQlIBzuXGulfMYvHeDRKU/m3oHfGCjLsH5tDo6lM2kh1Tte4emG\nG7l2eizzksPw8tJVPgpOVDby/uEy9ubXUN/STkNLB/nVTZTWtSICMxNCmZscRmyonerGNv74US5b\nj5bz2O2zSQjzpyz/OGbDt3iy7mNsXgYSFsOV/8V3Iifwr5emEup/nrP1hLnWH6UGkHsH/oZSaK7G\nRE3l3YPFHC5O4986nmPrJ5/w7LZYooP9mJ00BgN0OgzGmLN+pHI/BdXNHCtrACAxzJ+IQF+C7N4s\nSI1g8cRIrpgSTWSfC7I3zorn26/sZ/kTH3GV1y7+x+cP+NDBpwn/wqUrHobwcV11o4LPcjFXqSHm\n3oHfeWH3mUw7vzi8m0ujFkH9c7y18DDb/SP5NLuU4tKjeIlg8zrNLojKLdlNC2Mc1YQ7qgm1tRE3\nPYbxSQlERESCj79134etEyQfKvOhskdjRyeLO+v4x1VV5OzZwrTiV6kbk0b5dU9zyaSZeq+IGvHc\nPPBbSzn/cMSPOxck89Pl0+BPf8B31zMs5hkWD3P31AghXlDvgKxzaxYATEPgovsIufonhHjrzF6N\nDu4d+EsP0egTTlVrCF9fMt5aUXHHC1beX3k2bzsExVh/fPyhrRFaaqG1DjpaoL0FOltP01jAHgL2\nMdYFWHvokHZdqQvl1oHflB3iUEciiyZEkhQeYBUGRUHQkmHtlxqB/IKsPyQMd0+UGnTuu47f4cBR\nepgD7fGsnO/iTTNKKeUBXAr8IrJMRDJFJEtEVvdzfoqIfCIirSLyrXNpO2CMgQ8fsx52AVBzAltn\nMwU+KVydFjNoX6uUUqPNWQO/iNiAJ4DrgDTgThFJ61OtCngIePQ82g6M5mraPl2DeWkl1JdSe8L6\nAZA0JR1fb/f9xUYppc6VKxFxAZBljMk2xrQBa4HlPSsYY8qMMTuA9nNtO1CavUP5QsPDtNRVkP/U\nzez+dDMAl126aDC+TimlRi1XAn8CkN/juADXr4C53FZEVonIThHZWV5e7uLHd/P19uLBu27jhfj/\nIqHxMItKXqDMFsP4xLhz/iyllHJnIyYHYoxZY4xJN8akR0Wd+9azNi/h8klRfHXVQ7Qt/QE+0klg\n4oxB6KlSSo1uriznLASSehwnOstccSFtz5v98n8Hby8C43WPE6WU6suVwL8DmCgiqVhBeyXwBRc/\n/0Lanj8RWPRvg/41Sik1Gp018BtjOkTkAeBdwAY8b4zJEJH7nOefFpFYYCcQAjhE5GEgzRhT11/b\nwRqMUkqps5ORuCNlenq62blz53B3QymlRg0R2WWMSXel7oi5uKuUUmpoaOBXSikPo4FfKaU8jAZ+\npZTyMBr4lVLKw2jgV0opDzMil3OKSDlw4jybRwIVA9id0cATxwyeOW5PHDN45rjPdcxjjTEu7Xcz\nIgP/hRCRna6uZXUXnjhm8Mxxe+KYwTPHPZhj1lSPUkp5GA38SinlYdwx8K8Z7g4MA08cM3jmuD1x\nzOCZ4x60Mbtdjl8ppdSZueOMXyml1Bm4TeAXkWUikikiWSKyerj7M1hEJElEtojIIRHJEJFvOsvD\nReQ9ETnmfA0b7r4ONBGxicgeEXnLeewJYx4jIutE5IiIHBaRi9193CLyb85/2wdF5CURsbvjmEXk\neREpE5GDPcpOO04R+a4zvmWKyLUX8t1uEfhFxAY8AVwHpAF3ikja8PZq0HQA/2GMSQMWAt9wjnU1\nsNkYMxHY7Dx2N98EDvc49oQx/wbYaIyZAszCGr/bjltEEoCHgHRjzHSs53isxD3H/CdgWZ+yfsfp\n/H98JTDN2eZJZ9w7L24R+IEFQJYxJtsY0wasBZYPc58GhTGm2Biz2/m+HisQJGCN98/Oan8Gbh6e\nHg4OEUkEbgCe7VHs7mMOBS4DngMwxrQZY2pw83FjPSDKX0S8gQCgCDccszFmK1DVp/h041wOrDXG\ntBpjcoAsrLh3Xtwl8CcA+T2OC5xlbk1EUoA5wHYgxhhT7DxVAsQMU7cGy+PAfwKOHmXuPuZUoBz4\nozPF9ayIBOLG4zbGFAKPAnlAMVBrjNmEG4+5j9ONc0BjnLsEfo8jIkHAq8DDxpi6nueMtVTLbZZr\nicjngDJjzK7T1XG3MTt5A3OBp4wxc4BG+qQ43G3czpz2cqwfevFAoIjc3bOOu435dAZznO4S+AuB\npB7Hic4ytyQiPlhB/6/GmNecxaUiEuc8HweUDVf/BsGlwE0ikouVxrtCRF7AvccM1qyuwBiz3Xm8\nDusHgTuP+yogxxhTboxpB14DLsG9x9zT6cY5oDHOXQL/DmCiiKSKiC/WRZA3hrlPg0JEBCvne9gY\n81iPU28AX3K+/xLw96Hu22AxxnzXGJNojEnB+m/7D2PM3bjxmAGMMSVAvohMdhZdCRzCvcedBywU\nkQDnv/Ursa5jufOYezrdON8AVoqIn4ikAhOBz877W4wxbvEHuB44ChwHvj/c/RnEcS7C+vVvP7DX\n+ed6IAJrFcAx4H0gfLj7OkjjXwK85Xzv9mMGZgM7nf+9XwfC3H3cwI+BI8BB4P8Bfu44ZuAlrOsY\n7Vi/3X3lTOMEvu+Mb5nAdRfy3XrnrlJKeRh3SfUopZRykQZ+pZTyMBr4lVLKw2jgV0opD6OBXyml\nPIwGfqWU8jAa+JVSysNo4FdKKQ/z/wEU/CoSFO5QSAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2a1839ecb00>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['training', 'validation'], loc = 'upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training and validation accuracy seems to improve after around 60 epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 0s     \n"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy:  0.4105\n"
     ]
    }
   ],
   "source": [
    "print('Test accuracy: ', results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Nonlinearity (Activation function)\n",
    "- Sigmoid functions suffer from gradient vanishing problem, making training slower\n",
    "- There are many choices apart from sigmoid and tanh; try many of them!\n",
    "    - **'relu'** (rectified linear unit) is one of the most popular ones\n",
    "    - **'selu'** (scaled exponential linear unit) is one of the most recent ones\n",
    "- Doc: https://keras.io/activations/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://cs231n.github.io/assets/nn1/sigmoid.jpeg\" style=\"width: 400px\"/>\n",
    "<center> **Sigmoid Activation Function** </center>\n",
    "<img src=\"http://cs231n.github.io/assets/nn1/relu.jpeg\" style=\"width: 400px\"/>\n",
    "<center> **Relu Activation Function** </center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def mlp_model():\n",
    "    model = Sequential()\n",
    "    \n",
    "    model.add(Dense(50, input_shape = (784, )))\n",
    "    model.add(Activation('relu'))    # use relu\n",
    "    model.add(Dense(50))\n",
    "    model.add(Activation('relu'))    # use relu\n",
    "    model.add(Dense(50))\n",
    "    model.add(Activation('relu'))    # use relu\n",
    "    model.add(Dense(50))\n",
    "    model.add(Activation('relu'))    # use relu\n",
    "    model.add(Dense(10))\n",
    "    model.add(Activation('softmax'))\n",
    "    \n",
    "    sgd = optimizers.SGD(lr = 0.001)\n",
    "    model.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = ['accuracy'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = mlp_model()\n",
    "history = model.fit(X_train, y_train, validation_split = 0.3, epochs = 100, verbose = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcXGWd7/HPr/Zek96ydBKSgCEJSYBASFhkURgMiOy7\njIAog8tV5s44onNnUK8zwx0ZrzqyDCDgVQQRQZYJ4BZEFEKIYckGCSFJL1k63em99nruH09Vd3V3\n9ZJ0darP6d/79epXdZ06deo51d3f+vVznvMcMcaglFLKXTyFboBSSqn803BXSikX0nBXSikX0nBX\nSikX0nBXSikX0nBXSikX0nBXSikX0nBXSikX0nBXSikX8hXqhaurq82cOXMK9fJKKeVI69at22+M\nqRluvYKF+5w5c3jjjTcK9fJKKeVIIrJzJOtpt4xSSrmQhrtSSrmQhrtSSrlQwfrcc4nH49TX1xOJ\nRArdFNcIhULMnDkTv99f6KYopQ6jYcNdRB4ELgD2GWMW53hcgO8D5wPdwA3GmL8cSmPq6+spKytj\nzpw52M2q0TDG0NzcTH19PXPnzi10c5RSh9FIumUeBlYO8fh5wLz0183APYfamEgkQlVVlQZ7nogI\nVVVV+p+QUhPQsOFujHkZaBlilYuA/2es14DJIjL9UBukwZ5f+n4qNTHlo899BlCXdb8+vWx3Hrat\nlMrBGEMyZYgnDfFUikgsSXskTls4TiSeIuDzEPJ58XqEaCJJNJEilkj1Ph9IGUMqZUgZ8HkEv9eD\nzysYk34s/Rr2lqzvTdbjg7cxs/2kMQx2NU8Ddp2UwQAeAa9HEJF02/q2IWVMT1v9XunzHgB4RfB6\n7D4k088V6bs8ZWyb7Ova9TAGj0fwiuDxDF4QJTNt7bdDkt6+1yODrpNt2ZxKzjh62POQRuWwHlAV\nkZuxXTccccQRh/OlR6S1tZWf/exnfP7znz+o551//vn87Gc/Y/LkyYOu88///M+cccYZnHPOOaNt\npjrMkilDS1esNzxjSbwewe/zYAwc6IqxvzNKazjeE2IpY2iPxGkPx2mPJHr+0FMp6I4n6YzE6Yom\nSRmD1yN4RIglU4RjSbpiCRJJGxBJYxDA0xNONsxiQ6WqOiwy/xQPdRnqwf5xvuXMoxwR7g3ArKz7\nM9PLBjDG3AfcB7Bs2bJxd2Xu1tZW7r777gHhnkgk8PkGf6tWrVo17La/9a1vjbp96uAYY+iIJtjf\nEaW5K0Z7OE5nNEFHJNET1O3hOPGk6VMJdkYTdEYSHOiOsbstwr6OKMnUwf+6Brweyov8lId8eLOq\nweKgj7Kgj5qyIB6Rnso04PNQ5PdRHPAS8HnwCD1VZKbCBvB7PQS8gs/r6Wl30O9lUvq1ivxeYskU\n0XiKRCpF0Ocl6PcQ8Hr6hI1HpOcrkUoRTxoSyVRPFZp5fW9mPY+tqjPVrUfs90P1/GWq2SHXSb+e\nID3/EaRM3yo+87oi9sM2kf6A83kEn9dW8kCfD8TMh+Zgyz1Z+5n5TyGZfv1h29pvh0zPfxh9211I\n+Qj3Z4AvishjwAqgzRjjyC6Z2267jffff5/jjz8ev99PKBSioqKCLVu28N5773HxxRdTV1dHJBLh\ny1/+MjfffDPQO5VCZ2cn5513Hh/+8If585//zIwZM3j66acpKirihhtu4IILLuDyyy9nzpw5XH/9\n9Tz77LPE43F+8YtfsGDBApqamrj22mtpbGzklFNO4Te/+Q3r1q2jurq6wO/M4WeM4UB3nKaOKEGf\nh+KA7WLY2x6lsTXM7rYwTR1R9nfFONAVA2zoeT3C/s4oDa1hGlvDROKDV7iZ8A14hXjKEE+m8IhQ\nFvJRGvQxudjPqUdVM31SiJqyIJOL/ZQX+Sn2e0mmbLgYoLI4QHVZkMlF/j4hHvR5Cv4HrkbO4xE8\nHNrPS8R+yIwnIxkK+ShwFlAtIvXA7YAfwBhzL7AKOwxyG3Yo5I35aNg3n93Ipsb2fGyqxzG15dz+\niUWDPn7HHXewYcMG3nzzTV566SU+/vGPs2HDhp5hhA8++CCVlZWEw2FOOukkLrvsMqqqqvpsY+vW\nrTz66KPcf//9XHnllfzyl7/kuuuuG/Ba1dXV/OUvf+Huu+/mzjvv5IEHHuCb3/wmH/3oR/na177G\nCy+8wI9+9KO87v94FIknae6KUdfSzYaGNjY2trNlTwd1Ld10RhNDPlcEqkoCVBQHAEikA7qqJMCC\naWV8dP4UppaHqC4LUFViw7kkaIO7POQn5NfwVe41bLgbY64Z5nEDfCFvLRpHli9f3md8+A9+8AOe\neuopAOrq6ti6deuAcJ87dy7HH388ACeeeCI7duzIue1LL720Z50nn3wSgFdeeaVn+ytXrqSioiKv\n+3M4RRNJmjttX/QH+7vY0NDGhoZ2dreF7QGwZIruWHJAgE8rD7Fwehkr5lYyq7KYqeVBYgm7biyR\nYmp5iBkVRdROClFVGuxTKSuleo2rM1SzDVVhHy4lJSU937/00kv89re/5dVXX6W4uJizzjor5/jx\nYDDY873X6yUcDufcdmY9r9dLIjF0hToeGWNo6ojy6vZmXn5vP3/atp+WdPeIwR70yxbweVg4vZwl\nMycTSPcTh/xeasqCVJUEmDYpxKLaSdSUBXO9nFLqII3bcC+EsrIyOjo6cj7W1tZGRUUFxcXFbNmy\nhddeey3vr3/aaafx+OOP89WvfpVf//rXHDhwIO+vcTCSKcPvNu/lqfUNxBIpioM+gj4P9Qe6eW9v\nZ0+YVxT7Oe1D1cyqLO55bknAS1WpDe5ZlcV8aEppz0EvpdTY03DPUlVVxWmnncbixYspKipi6tSp\nPY+tXLmSe++9l4ULFzJ//nxOPvnkvL/+7bffzjXXXMNPfvITTjnlFKZNm0ZZWVneXyeXA10x3t3b\nQVs4TmckQUNrmJ+vraOhNcyUsiDVpUHC8SThWJJpk0Kce8xU5k8rY9nsShbVlg85NlgpdfjJUAPt\nx9KyZctM/4t1bN68mYULFxakPeNBNBrF6/Xi8/l49dVX+dznPsebb7456u1u3ryZ+fMX0NIdY09b\nhMbWcM9okp3N3WxsbKehdWD30Yq5ldxw6hz+6pip+LTqVmpcEJF1xphlw62nlfs4smvXLq688kpS\nqRSBQID777//kLeVMoZw+oBlU0eUi/7phQEnvgR9HmZWFHHC7Ao+dcpsFk4vp7IkQFnIjiapKAmM\ndpeUUgWi4T6OzJs3j/Xr1x/Sc1PG0BlJ0BlN0B1LEokne07GMBg+dcrs9OiTENMn2REnVSUBHQqo\nlEtpuDtUMmV65gzpTJ9xmUwZPCIU+b1UlgQoDngpDfrY2hHify2fuN1dSk1EGu4OkUim6IrZOUk6\nook+k0B5PUJ5yM+kYj+lQV/PKddKqYlLw30cysxv0hGOE0lX55m5TTwilAZ9VBYHCPq9BH2e9Dwk\nGuhKqV4a7uOEMYb2SIIDXTE6onYWQa/HdrFMLvIT8HkpCngpDng1yJVSw9LxbaNQWloKQGNjI5df\nfnnOdc466yz6D/nMFokn+da/fYf12/eys7mLcDzJrTdcSaUvzsLp5RxZU8qMimJqyoLa5aKUGjGt\n3POgtraWJ554YkTrJlOZObuT6QsrJPmvu/+TCy69kjm1VZSFfPzuNy+OcYuVUm6nlXuW2267jbvu\nuqvn/je+8Q2+/e1vc/bZZ3PCCSewZMkSnn766QHP27FjB4sX22uHh8Nhrr76ahYuXMgll1xCOBwm\nmUrR1BHlmus/w7HHn8hxxx3L7bffjkeE5x59kKa9e7j+8gu46PxzERHmzJnD/v37Afjud7/L4sWL\nWbx4Md/73vd6Xm/hwoV89rOfZdGiRZx77rmDzmGjlJqYxm/l/vxtsOed/G5z2hI4745BH77qqqu4\n9dZb+cIX7CSXjz/+OC+++CJf+tKXKC8vZ//+/Zx88slceOGFg44Pv+eeeyguLmbz5s28sW49J684\niR37uylqC/P3X/9nZk6fQtALF52/ku492/n6V/4n9931A1avXj1g3vZ169bx0EMPsWbNGowxrFix\ngjPPPJOKiooRTy2slJqYtHLPsnTpUvbt20djYyNvvfUWFRUVTJs2ja9//esce+yxnHPOOTQ0NLB3\n795Bt/Hyyy9z2ZVXs2N/F4Gpc5m3cBElQR/zppSydvUqzjvrVM44ZTmbNm1k06ZNQ7bnlVde4ZJL\nLqGkpITS0lIuvfRS/vjHPwIjn1pYKTUxjd/KfYgKeyxdccUVPPHEE+zZs4errrqKRx55hKamJtat\nW4ff72fOnDk5p/oFaA/H6YomaGyLMDeWZGp5iJDfy7RJIfY01HHnnXeydu1aKioquOGGGwbdzkiM\ndGphpdTEpJV7P1dddRWPPfYYTzzxBFdccQVtbW1MmTIFv9/P6tWr2blz54DnhGP2pKIdzV2cuOJU\n/vD8UyyYVkbTrm288/bbALS3t1NSUsKkSZPYu3cvzz//fM/zB5tq+PTTT+dXv/oV3d3ddHV18dRT\nT3H66aeP3c4rpVxj/FbuBbJo0SI6OjqYMWMG06dP55Of/CSf+MQnWLJkCcuWLWPBggV91t/XHmFn\nczcGmFlRxD/9w63c9OlPs2jRMSxcuJATTzwRgOOOO46lS5eyYMECZs2axWmnndazjZtvvpmVK1dS\nW1vL6tWre5afcMIJ3HDDDSxfvhyAz3zmMyxdulS7YJRSw9Ipfw+BMYZIPMXutjCd0QSTi/zUVhTh\n84zPf4Sc8r4qpYanU/6OgXgyRXNXjLbuONFEEo8IMyuKqSj26+yKSqlxRcN9hOLJFO83dRJPpCgJ\n+qguLWJSkV8vYqGUGpfGXbgbY8ZdFZxIpvigqYtE0nBUTSnFwXH3tg2qUN1uSqnCGldlZygUorm5\neVwFUjKV4oP9XcSSKeZUlTgu2JubmwmFQoVuilLqMBtXSTVz5kzq6+tpamoqdFMAe3Wj5s4YsUSK\nqtIAde3eQjfpoIVCIWbOnFnoZiilDrNxFe5+v5+5c+cWuhkAdEYT3PDg66yva+U/r1nK0iXTC90k\npZQasXHVLTNedEYT3PhQb7Cfr8GulHKYcVW5jwdt4Tg3PbyW9XWt/OBqDXallDNpuGfZ3xnlUz96\nna37OrRiV0o5moZ7WmNrmOseWENjW5gHrj+JM4+uKXSTlFLqkGm4A3vaIlx136u0dsX5yU0rOGlO\nZaGbpJRSozLhD6ge6Irx1z9aQ0tnjJ9+RoNdTUCxrkK3QI2BCV25d0YT3PDQ6+xs6ebHNy7nuFmT\nC90kNd4ZY78GmyQuHoFwC0TawBeCYJn98gVzr5/ZZlsdiBdKp4DXf3BtSqXs7cFMXJdKwdYX4bV7\n4IM/QO1SWHYTLL4MPD5o2Q4HdkBxJVR9yN4CRDuhc69tc7AUAiXQ9C5s+y1s+x3Eu+361fNg8mz7\nvKJK8Ifsc2Od6dsOe2uSMGmWXbekCrr2Q8ce6N4PJr1f4oXJR9htlk23V2jb8UeoX2ufO2sFzDzJ\nvuf7NsHejX0/sALFtg3FVSAe24ZMO6Id9nuTsvsSKLXvf6aNqSQUVdj98JdApBW6m+32S6pte4qr\nIRGx24l3g6/Ivjf+kt7l0U4IH7DP7W6GYy6EEz51cD/ngzRhwz2ZMtzyk3VsaGznv647kVOOqip0\nk9wp3Ap7N8D+rdC8zf5RlE2HsmngL+79hU/GoHSqXV46NR0cpfaPsWW7fW7HbqieDzNOgMojYf97\nULcG9myA2uNh/vn2jzDWDZufhXd+YYM2Q7w2ZL1++4c+eTZUzLaBvHeDDYZU0m6rdqkN5cY3oXG9\nbUO03f6Rigeqj4api6C8Flp3wv5tNgxjA+flB2wAVMy2YRQsBW8QPF77vjSut6FhGwklNeAN2Pck\nGbUhUTbNvm8i9n3o2GPfu2QMUgn7VH+xfc98ofRzY7atlUfaYCyvte91xx67v627oHwGnPwFeP93\n8MwXYdVX7GtmgjWjqBKS8cH3D7E/l/IZsOdt+/6b5Ch+cQYjQPoM9smzYetv4LW7+67i8dn3Aey6\nsW5IxXNsygOBMvvzEG/vB04qYX/2gRK7rfABG9CZ1y+qsI917rPv1UgFStMfMpWQOIjnHaJxNeXv\n4XTvH97njue38G+XLuGa5UcUrB0FZQy01cPkWcOv21YP7zwBW56zv/hTF8HUxTD/PAhN6rtu8/uw\n4UlbzdWv7f0j94XsH0V388Dte3y9ITWorD9s8fbdbiJitzFrha3sou32j796Xu++miQk0oHZ2QTt\n9b0hVlRh9wdg91v2+QAeP0xbDDULIFhu259KQNMWWyF27LZVZdU8qDrKBnNxpV03EbWhEGm3lXnr\nLnsb67ZtSCZs4NcuhenH2bDp2GO3mUrYgPcG7Adix277ZQyUT7dBX1RhH/cF7fJMRZqI2g8wb9AG\nfMt2+yHSucd+qJVNtx8yx14JCz9h1zUGdr0KG5+yP8+qeVA5F7pboDn9wewL9X4wi6e36i2fAUd+\nxFbeGYmYfb3uFvvzTkR7P7CDZenbUvszbauDAztttV4yxW6/pMZ++EF6Hz6w7WitgynHwNzT7XqJ\nmP0waViX/hkusm33BXrbYoxta3ez/f3JBLovZD8s+zNm4PJE1P4cQpN622VMb3HiL7K/G/5i+7sY\nzVTxQbuvgdK+bRqFkU75OyHDfWNjGxff9SfOXjCVe647YdxNVHbYbP0tPHIZXPYjWHJ57/L23fD8\nV6D7gL0f77IVLAZqT7AhuW+LDaiahXD9s1CaHl20aw389FL7R1+7FD50Dhxxsq10y2faroNEzP5r\nH++2lUxRhQ2LcIsNsM59vf/KpuJQMdeGdMkUW603rre31UfbbVceaZdt+pXtGpi2BJZeB0ecOnRX\nRTIO7Q02IDNVMdgui5bttg1TFg7dpZJKHVx3SCGlkr3BpBxLw30QkXiSC3/4Cge647x46xlUluTn\n03RMGGOr4B0vw45X0tVNs62GQuW93QehybZroHWXrSw+/LdQMcduIxGDv/zYdhmc++2+Fclr98IL\nX7VVxc0v2QCNdcFD59lKr/YEu54IzDndfgBUHWWXJRPw/u/h8U/Z6vP6Z20g/vQy263y10/Z5Uqp\nvNKLdQzi3194l/f2dvLwjSeNj2Bvfh82Pmmr0w+dA0WTbT/1m4/A6/fDgQ/semXTbddAxRz7b393\ns61WNz9rH/cGYNJMaG+E9Y/ASTfZf/X/8H9ssIMN/ZLq3tfu2G27HXxBePx6uOnX8ORnbbfGNY/B\n0R8bvN1eHxx9LnzyF/CzK+HBlbYaL5sG1z9nuw6UUgUzocJ9275OHvrzB1x38hGcNX9KYRuz5b9h\nzX/ZkQoZHh/MWGbDNd5luxVO/R8w90xbMefqPgofgHgYSqfZ7oG2Bhvor99vu0+mLoEVn4M199gu\niD7hvsd+aFzwf233zD2n2Or/vH8fOtizzT3dBvwjV9i+1xueswGvlCqoEYW7iKwEvg94gQeMMXf0\ne7wCeBA4CogAnzbGbMhzW0fth7/fSsjn5W/PObqwDXn9flj19/ag1kf/Fxz/SXug6N1VsH21HSa1\n4hbb7TKcogr7lTFpBlz4Azj1S7ar5siP2Ap/zT22L336cb3rdjTaCnveOXD638Mf74TlfwMr/ubg\n9mfOh+GLb9iuomDZwT1XKTUmhg13EfECdwF/BdQDa0XkGWPMpqzVvg68aYy5REQWpNc/eywafKi2\nN3XyzFuNfOb0I6kqHeIA2WikkrDuIVj3cHpERMx2eZzzDTsqAewBv+e/aoftXfkT270BdpjaESuA\nb+anLdUfsl/Q20XS3tB3nY499oAhwEe+bruFZi0/tNebNOPQnqeUGhMjOcy/HNhmjNlujIkBjwEX\n9VvnGOD3AMaYLcAcEZma15aO0g9XbyPg8/DZ048cmxdoXA8PnAP//Xe2/7v2eHsQ0lcEP7/Ojh/e\n8w784kYbqJfe1xvsY610qh062LG77/JMtwzYURSzT9HRFEq5xEjSZQZQl3W/HljRb523gEuBP4rI\ncmA2MBPYm49GjtaO/V08/WYjN546h5qyPFftxtjujN//ix2be+kDdlRJpn88EYPffgNeu8t2x5RU\nwzWPHt7uC4/X9oO3N/Yui3basdzaP66UK+WrdLwD+L6IvAm8A6wHBpyeJiI3AzcDHHHE4Ttx6Ier\nt+HzCDefmeeqPRGD5261I1uWXAEf/4+BJ/T4ArDyX2HuGfZD4GP/ak96OdzKpvcN9869vcuVUq4z\nknBvALJPYZyZXtbDGNMO3Agg9oygD4Dt/TdkjLkPuA/sOPdDa/LBeb+pk6fWN/CpU2YzpWyUF4re\n/geoe90OVyyuhHU/tqNdzrwNzrot92iWjPkr7VehlNfaOUAyMkGvlbtSrjSScF8LzBORudhQvxq4\nNnsFEZkMdKf75D8DvJwO/IK74/ktFPm9fOEjHzr0jTSut10r21/qu9zjg4vvgeOvzfWs8aW8tm/7\nO/bY27LagjRHKTW2hg13Y0xCRL4IvIgdCvmgMWajiNySfvxeYCHwYxExwEbgpjFs84i9/kELv9m0\nl698bD7VhzpCJjNssajSdqmccL09bb67xfabO2WUSHmt7WOPtNshi5mDq1q5K+VKI+pzN8asAlb1\nW3Zv1vevAgUePN6XMYZ/WbWZaeUhPn3a3EPf0Hsv2ilMP/v73v70YKmdmtVJMhV6x+50uO+xsw3q\nuHSlXMkhMx4dvOfe3s1bda383blHUxQYxfC+cIudXbD/gVKnKU+He6avvWN3ena/CTppmlIu58pw\njydT/PuLW1gwrYxLT5g5uo11t/Q9A9Spek5kyoT7Hh0po5SLuTLc137QQl1LmC+fPQ+vZ5SVafhA\n71VonKynW6ax91b725VyLVeG++p39xHwejjj6Jq+DyRi8OI/ws5XR7ahVNJeuqvIBeHuD9n9aG+0\nJ1517NGZG5VyMVfOCrn63SZWHFlJSTBr95IJ+OVNsPkZO13u7FOG31C4FTDu6JYBO2tj+257SbdE\nRLtllHIx11XudS3dbNvX2XdK31QKnv68DfbQJNi3eWQbC6evROSGbhmwlXp7Q9YYd+2WUcqtXBfu\nL73XBMBH5md1yTz/FXj753Z63eOutZdoS6UG2UKWzMWV3dAtA3bETOZanKCVu1Iu5r5w37KPIyqL\nmVtdYhd0t8DaB+DEG+GMr8CUBfYkpNadw2+sOxPuLumWKauFriZ7uT7Qyl0pF3NVuEfiSf70/n4+\nMr+m96LXbfX29sgz7W1Nev7ypi3DbzBTuRe7JNwzY90b19tbrdyVci1XhfuaD1qIxFOctSCrvz1z\ngYpJ6bnPaubb2xGFe7rP3TXdMukwb/iLvai2v6iw7VFKjRlXhftL7+4j6PNwypFVvQszlXt5eg6Y\nosm2e2LfCMK9uwXEA8Hy/De2EDLvwb5NWrUr5XIuC/cmTj2qipA/a7qB9gY7e2P2XDBTFkDTCEbM\nhNNnp3pc8jZlAt0ktb9dKZdzSWrZIZAf7O/qOwQSoK3BVurZl4+rWQhNIxgxEz7gni4ZsMNA/ekD\nzVq5K+Vqrgn37fu7AFg4vV8XSnvDwGl5pyyARBhadwy90e4W94xxBztJWKbfXSt3pVzNNeG+uzUM\nwPRJ/a621Fbf29eckRkxM1y/e9glk4Zly4yYKdeLdCjlZq4J98a2CCIwLTvcUyk7l8qkfjND9oyY\nGabfPdzqrm4Z6J1ATCt3pVzNNeG+uzXMlLIgfm/WLnXtg1R8YLiHyqF85vCVu9u6ZaC3Ytc+d6Vc\nzT3h3hZh+qR+47bb0mPc+3fLwPAjZhJRiHfZoZNuUjnXDu/s/4GnlHIV18wK2dgWZsG0fpeMa0+P\ncc91ndOaBbDjFTutryfHlZrcdgJTxrFXw7Ql2i2jlMu5onI3xtDYGh6ics9RpU5ZaKe9PbAj90Yz\n88q4rVvGF4DapYVuhVJqjLki3Fu740TiqYEjZdobwBfKHdA9I2YG6ZoJu2zSMKXUhOKKcG9ss8Mg\nayf3r9zrbd9yrotADzdixq3dMkqpCcEV4b67NQLkGOPe3pD7YCpAsNQ+1rw99+Nu7ZZRSk0I7gj3\nQSv3hqFHhZTU2PnNc9FuGaWUg7ki3BvbIvg8QnVpsHdhMg6dewav3AFKqqF7f+7HwgfAGwR/cX4b\nq5RSh4Erwn13a5ip5SG8nqy+9Y7dYFK5h0FmFFdDV3PuxzInMOXqr1dKqXHOFeHe2BZhRq4uGcg9\nDDJjuMpdu2SUUg7linDf3RZm+uQcB1NhmMq9yl5PNdY98DG3TferlJpQHB/uqZRhT86pB/pdgSmX\nkmp7m6t6725xz7VTlVITjuPDfX9nlHjSUJurcg9OspOEDaY4He5dOcLdjdP9KqUmDMeHe2NbZox7\nrmGQQ1TtkFW59zuoaox2yyilHM3x4T7oRTrac1yko79MuPcf6x7rgmRMT2BSSjmW48M9U7nnPoFp\nmHAfrFtGT2BSSjmc48N9d2uYoM9DRbG/d2E8bA+SDjUMEiBYBt7AwAOqOq+MUsrhnB/ubRFqJxch\n2ScbNW+zt5Vzh36ySO4TmXReGaWUwzk+3BvbwgP723e9Zm9nnjT8BkqqclTu2i2jlHI2x4f77tbI\nwP72ujX2GqGTjxh+A8XVA/vcM5W7dssopRxqROEuIitF5F0R2SYit+V4fJKIPCsib4nIRhG5Mf9N\nHSieTLG3I0LtgMp9DcxaPrJ5YXJNQRButbdauSulHGrYcBcRL3AXcB5wDHCNiBzTb7UvAJuMMccB\nZwH/ISKBPLd1gL3tEYyB6dmVe3sjtO2CWSePbCO5+tzDLRAos5ekU0opBxpJ5b4c2GaM2W6MiQGP\nARf1W8cAZWKPapYCLUAiry3NYU96GOS07Mq9bo29PWLFyDZSUgWxDkhEe5d169mpSilnG0m4zwDq\nsu7Xp5dl+yGwEGgE3gG+bIxJ5aWFQ+iM2s+P8lDWMMhda+wc7NOOHdlGco11Dx/QeWWUUo6WrwOq\nHwPeBGqB44EfisiASV1E5GYReUNE3mhqGuQKSAchmrCfH0Ff1m7UvQYzTgSvf5Bn9ZNr8rDWncOf\n3aqUUuMv4HlpAAANYElEQVTYSMK9AZiVdX9melm2G4EnjbUN+ABY0H9Dxpj7jDHLjDHLampqDrXN\nPWL9wz3WBbvftgdTR6p/5Z6M23HyNQOar5RSjjGScF8LzBORuemDpFcDz/RbZxdwNoCITAXmA4Nc\neTp/MpV7IBPuDevAJEd+MBWy5pdJh3vLdkglNNyVUo7mG24FY0xCRL4IvAh4gQeNMRtF5Jb04/cC\n/xt4WETeAQT4qjFmkEsc5U9v5e61C3alD6bOGsHJSxnFVfY20y3TtMXe1szPQwuVUqowhg13AGPM\nKmBVv2X3Zn3fCJyb36YNL5ZIAlmVe90aqFl4cCNdQpNBvL2Ve9O7gED10fltrFJKHUaOPkO1zwHV\nVArqXx/5EMgMj6fviUxNW+yZrYHiPLdWKaUOH0eHeyy7z33TryDSdnD97RnZJzI1vav97Uopx3N0\nuEcTKUTAV/8aPHULzFwOiy4++A1lJg9LJmD/Vu1vV0o5nqPDPZZMcYyvEXn0atuVcu3PwV80/BP7\ny0we1roTklGt3JVSjufocPdEWnnA+2/gC8F1vzz0+dczfe49I2U03JVSzjai0TLj1ZSODUynGS58\nHCpmH/qGiqttf/2eDfZ+jY6UUUo5m6Mrd+J24jDKpo9uOyXpse47/2QvzRcsG932lFKqwBwd7pII\n22/8oxy2mJmCoO51PZiqlHIFR4e7pyfcQ0OvOJzMFASJsPa3K6VcwdnhnkzPwZ6vyh20cldKuYKj\nw92bqdx9earcQSt3pZQrODvck+kDqqMN96IK7HxnaOWulHIFRw+F9KYiRCVI0DPKzyiP146R9wag\naHJ+GqeUUgXk6HD3JyPEJUgwHxsrnQalU/KxJaWUKjhHh7svFSXuyUu0w8V3gb8kP9tSSqkCc3S4\n+02UhDdP4V67ND/bUUqpccDRB1QDqQgJzygPpiqllAs5Otz9JkbCq+GulFL9OTrcgyZKUsNdKaUG\ncGy4p1KGIFFSGu5KKTWAY8M9lkxRRIzUaE9gUkopF3JsuEcTKUISI+U7hCsvKaWUyzk43JOEiGE0\n3JVSagDHhnsskaKIKGi4K6XUAI4N92jcVu6HdEFspZRyOceGezwWwStm9BfqUEopF3JsuCci3QBI\nYJQX6lBKKRdybLjHo+lwH+1VmJRSyoUcG+7JaBcAnoD2uSulVH+ODfdEunL3areMUkoN4NhwT8Vs\nuHuCGu5KKdWfc8M9Xbn7NNyVUmoA54Z7PAyAN6hXT1JKqf4cG+4mbit3v1buSik1gHPDPd3n7g9p\n5a6UUv05NtxJRADwh7RyV0qp/pwb7jHb5x7Qyl0ppQZwbLhL0lbunoCGu1JK9TeicBeRlSLyrohs\nE5Hbcjz+FRF5M/21QUSSIlKZ/+b28iTCJI2A1z+WL6OUUo40bLiLiBe4CzgPOAa4RkSOyV7HGPMd\nY8zxxpjjga8BfzDGtIxFgzM8iTARCYLIWL6MUko50kgq9+XANmPMdmNMDHgMuGiI9a8BHs1H44bi\nSUaIEhjrl1FKKUcaSbjPAOqy7tenlw0gIsXASuCXo2/a0LyJCFEJjvXLKKWUI+X7gOongD8N1iUj\nIjeLyBsi8kZTU9OoXsibihATvVCHUkrlMpJwbwBmZd2fmV6Wy9UM0SVjjLnPGLPMGLOspqZm5K3M\nwZeMEBftllFKqVxGEu5rgXkiMldEAtgAf6b/SiIyCTgTeDq/TczNl4oQ82jlrpRSufiGW8EYkxCR\nLwIvAl7gQWPMRhG5Jf34velVLwF+bYzpGrPWZvGnokQ8enaqUkrlMmy4AxhjVgGr+i27t9/9h4GH\n89Ww4fhTUTp8YzqUXimlHMuxZ6j6TZSEdssopVROjg33oImS9OpQSKWUysXh4a6Vu1JK5eLYcA8Q\nI+UrKnQzlFJqXHJmuKdShIhhtHJXSqmcnBnu6Qt1pPxauSulVC6ODPfM9VPxaeWulFK5ODLcE1F7\nnpTRPnellMrJkeEej6Qrd7+eoaqUUrk4NNxt5S5+7ZZRSqlcHBnuiXTlLnr9VKWUysmZ4R7NVO7a\n566UUrk4MtyTUVu5ewLa566UUrk4MtwTMRvu3qCGu1JK5eLIcE9lwj2g3TJKKZWLs8M9qAdUlVIq\nF0eGu4mFAfCFtFtGKaVycWa4p6cf8Gufu1JK5eTIcCceIWp8BAOBQrdEKaXGJYeGe5goAQI+ZzZf\nKaXGmjPTMREmTICgz1voliil1LjkyHCXeJiI0cpdKaUG48h09CTDhAkS1HBXSqmcHJmOnkSUiPa5\nK6XUoByZjp5kmAgBfB4pdFOUUmpccmS4+5IRohJERMNdKaVycWS4e5MR4qJj3JVSajCODHdfKkJM\n9CpMSik1GIeGe5SYR8NdKaUG48hw96eiJD3BQjdDKaXGLUeGeyAVIa6Vu1JKDcp54Z6M4yVJ0quV\nu1JKDcZ54R63c7knvXoVJqWUGoyDw127ZZRSajDOC/eEDXej4a6UUoNyXrhnKnefdssopdRgHBvu\n+LRyV0qpwYwo3EVkpYi8KyLbROS2QdY5S0TeFJGNIvKH/DYzSzrcjV8rd6WUGoxvuBVExAvcBfwV\nUA+sFZFnjDGbstaZDNwNrDTG7BKRKWPVYA13pZQa3kgq9+XANmPMdmNMDHgMuKjfOtcCTxpjdgEY\nY/blt5lZ0gdURcNdKaUGNZJwnwHUZd2vTy/LdjRQISIvicg6EflUvho4QFEFf0otIRGcPGYvoZRS\nTpevA6o+4ETg48DHgH8SkaP7ryQiN4vIGyLyRlNT0yG9UOqI0/hk7GvESmpH1WCllHKzkYR7AzAr\n6/7M9LJs9cCLxpguY8x+4GXguP4bMsbcZ4xZZoxZVlNTc0gNjiVTAHqJPaWUGsJIEnItME9E5opI\nALgaeKbfOk8DHxYRn4gUAyuAzfltqhVN2HAP+rxjsXmllHKFYUfLGGMSIvJF4EXACzxojNkoIrek\nH7/XGLNZRF4A3gZSwAPGmA1j0eBYQit3pZQazrDhDmCMWQWs6rfs3n73vwN8J39Nyy3TLRP0argr\npdRgHJeQ0XgSgKDfcU1XSqnDxnEJ2XNAVSt3pZQalOMSMhpPd8to5a6UUoNyXEL2Vu46WkYppQbj\nvHBPaOWulFLDcVxCRhP2gKr2uSul1OAcl5A6zl0ppYbnuISsKQty/pJpTC72F7opSik1bo3oJKbx\n5MTZlZw4u7LQzVBKqXHNcZW7Ukqp4Wm4K6WUC2m4K6WUC2m4K6WUC2m4K6WUC2m4K6WUC2m4K6WU\nC2m4K6WUC4kxpjAvLNIE7DzEp1cD+/PYHKeYiPs9EfcZJuZ+T8R9hoPf79nGmJrhVipYuI+GiLxh\njFlW6HYcbhNxvyfiPsPE3O+JuM8wdvut3TJKKeVCGu5KKeVCTg33+wrdgAKZiPs9EfcZJuZ+T8R9\nhjHab0f2uSullBqaUyt3pZRSQ3BcuIvIShF5V0S2ichthW7PWBCRWSKyWkQ2ichGEflyenmliPxG\nRLambysK3dZ8ExGviKwXkefS9yfCPk8WkSdEZIuIbBaRUybIfv9t+vd7g4g8KiIht+23iDwoIvtE\nZEPWskH3UUS+ls62d0XkY6N5bUeFu4h4gbuA84BjgGtE5JjCtmpMJIC/M8YcA5wMfCG9n7cBvzPG\nzAN+l77vNl8GNmfdnwj7/H3gBWPMAuA47P67er9FZAbwJWCZMWYx4AWuxn37/TCwst+ynPuY/hu/\nGliUfs7d6cw7JI4Kd2A5sM0Ys90YEwMeAy4qcJvyzhiz2xjzl/T3Hdg/9hnYff1xerUfAxcXpoVj\nQ0RmAh8HHsha7PZ9ngScAfwIwBgTM8a04vL9TvMBRSLiA4qBRly238aYl4GWfosH28eLgMeMMVFj\nzAfANmzmHRKnhfsMoC7rfn16mWuJyBxgKbAGmGqM2Z1+aA8wtUDNGivfA/4BSGUtc/s+zwWagIfS\n3VEPiEgJLt9vY0wDcCewC9gNtBljfo3L9zttsH3Ma745LdwnFBEpBX4J3GqMac9+zNhhTq4Z6iQi\nFwD7jDHrBlvHbfuc5gNOAO4xxiwFuujXFeHG/U73M1+E/XCrBUpE5Lrsddy43/2N5T46LdwbgFlZ\n92eml7mOiPixwf6IMebJ9OK9IjI9/fh0YF+h2jcGTgMuFJEd2O62j4rIT3H3PoOtzuqNMWvS95/A\nhr3b9/sc4ANjTJMxJg48CZyK+/cbBt/HvOab08J9LTBPROaKSAB78OGZArcp70REsH2wm40x3816\n6Bng+vT31wNPH+62jRVjzNeMMTONMXOwP9ffG2Ouw8X7DGCM2QPUicj89KKzgU24fL+x3TEni0hx\n+vf9bOyxJbfvNwy+j88AV4tIUETmAvOA1w/5VYwxjvoCzgfeA94H/rHQ7Rmjffww9l+1t4E301/n\nA1XYo+tbgd8ClYVu6xjt/1nAc+nvXb/PwPHAG+mf96+Aigmy398EtgAbgJ8AQbftN/Ao9phCHPtf\n2k1D7SPwj+lsexc4bzSvrWeoKqWUCzmtW0YppdQIaLgrpZQLabgrpZQLabgrpZQLabgrpZQLabgr\npZQLabgrpZQLabgrpZQL/X9LA+0gjQdGGAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1f06255c3c8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['training', 'validation'], loc = 'upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training and validation accuracy improve instantaneously, but reach a plateau after around 30 epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 9376/10000 [===========================>..] - ETA: 0s"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy:  0.9208\n"
     ]
    }
   ],
   "source": [
    "print('Test accuracy: ', results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Optimizers\n",
    "- Many variants of SGD are proposed and employed nowadays\n",
    "- One of the most popular ones are Adam (Adaptive Moment Estimation)\n",
    "- Doc: https://keras.io/optimizers/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://cs231n.github.io/assets/nn3/opt2.gif\" style=\"width: 400px\"/>\n",
    "<br><center> **Relative convergence speed of different optimizers** </center></br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def mlp_model():\n",
    "    model = Sequential()\n",
    "    \n",
    "    model.add(Dense(50, input_shape = (784, )))\n",
    "    model.add(Activation('sigmoid'))    \n",
    "    model.add(Dense(50))\n",
    "    model.add(Activation('sigmoid'))  \n",
    "    model.add(Dense(50))\n",
    "    model.add(Activation('sigmoid'))    \n",
    "    model.add(Dense(50))\n",
    "    model.add(Activation('sigmoid'))    \n",
    "    model.add(Dense(10))\n",
    "    model.add(Activation('softmax'))\n",
    "    \n",
    "    adam = optimizers.Adam(lr = 0.001)                     # use Adam optimizer\n",
    "    model.compile(optimizer = adam, loss = 'categorical_crossentropy', metrics = ['accuracy'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = mlp_model()\n",
    "history = model.fit(X_train, y_train, validation_split = 0.3, epochs = 100, verbose = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VPW9//HXZ5Ysk30lJAHCvskimyLuK664i9W6tL38\ntPaqXcW2t63V9npvrbW2UkqV1t2rqHUpiitSFZR9X0IgkIQlCyF7Zv3+/jgDhJBNmTCZyef5eORB\n5szJme93Qt755nO+53vEGINSSqnoYgt3A5RSSoWehrtSSkUhDXellIpCGu5KKRWFNNyVUioKabgr\npVQU0nBXSqkopOGulFJRSMNdKaWikCNcL5yZmWkKCgrC9fJKKRWRVq5cWWmMyepsv7CFe0FBAStW\nrAjXyyulVEQSkV1d2U/LMkopFYU03JVSKgppuCulVBQKW829LV6vl9LSUpqbm8PdlKgRFxdHfn4+\nTqcz3E1RSp1APSrcS0tLSUpKoqCgABEJd3MinjGGqqoqSktLGThwYLibo5Q6gXpUWaa5uZmMjAwN\n9hARETIyMvQvIaV6oR4V7oAGe4jp+6lU79SjyjJKKRUNmr1+Xl1VSlZiLKcMyiAl/sSf89Jwb+Hg\nwYO88MILfPe73/1KX3fJJZfwwgsvkJqa2u4+v/jFLzjzzDM5//zzj7eZSqkTwBjDwUYvB5u81DR5\n8fkD9E93kZUU2+FfxEUV9dz1/Cq27KsDwCYwOjeFlHgnHn8Arz/AVSfnccvUgm5tv4Z7CwcPHmTO\nnDnHhLvP58PhaP+tWrhwYafH/vWvf33c7VMqWhljqG3yEeu0EeuwHQ5Prz+Az2+Ij7GfsLY0e/28\nvW4vT39ezPqymmOeT4ixk5/mItZpw24TYh02BmclMqJvMsYYHn5nC3FOO0/eMomkOAefFVWxfOcB\nGj0+nHYbibEO4hzd3x8N9xZmz55NUVER48ePx+l0EhcXR1paGlu2bGHbtm1ceeWVlJSU0NzczD33\n3MOsWbOAI0sp1NfXc/HFF3P66afz+eefk5eXxxtvvEF8fDy33XYbl112Gddeey0FBQXceuutvPXW\nW3i9Xl555RVGjBhBRUUF3/jGN9izZw9Tp07l/fffZ+XKlWRmZob5nVEqdOqavZQdbKKsuont5fWs\n2l3Nyl0Hqax3A+CwCXFOO81eP76AASAzMZbhOYkMzkrEJoLbF8DjCxAfYyMl3klKvJNRfVOYVJBG\nnPPY4DTGsGBlKTsqG8hLjScvLZ6ROcnkpMQd3qemycvTnxfz9OfFVDV4GJKdyH3TR9AnOZaUeCc2\nm1ByoJEdFQ2UHWzC5w/gCxga3D7eWruH57/YDcCUgnQev/Hkw8c+ZVBGd7+lbeqx4f7AWxvZtKc2\npMcclZvMLy8f3e7zDz/8MBs2bGDNmjUsXryYSy+9lA0bNhyeRjh//nzS09Npampi8uTJXHPNNWRk\nHP2NKyws5MUXX+Rvf/sb119/Pa+++io333zzMa+VmZnJqlWrmDNnDo888ghPPvkkDzzwAOeeey73\n338/7777Lk899VRI+69UZ7z+AA6bHFV2qKx3s76sBrsI2cmxZCfFkeZydulkvT9g+OfqMj7csp+S\nA02UVDdysNF71D4DMlycOSyTkTnJePwBGj0+mjwB4pw2XDF2RITiyga2ldfz+uoyBIh12omx22j0\n+Kht9uEP/hKIsduYMCCVS8f05aoJ+STGOjjQ4OEnC9byweZybALBXQEYl5/ChaNzaPL4efrzYurc\nPs4dkc23Tx/IaYO7PnPPGMPemmb21jQzLj8Fh73VXJVAAGwndv5Kjw33nmDKlClHzQ9//PHHef31\n1wEoKSmhsLDwmHAfOHAg48ePB2DixIkUFxe3eeyrr7768D6vvfYaAJ9++unh40+fPp20tLSQ9kep\n9hhjeG7ZLn6zcDMAeanx9EmOY1dVI2UHm47Z/9wR2fzhhvHtnig0xrBo435+/95WdpTXMCrVR1p2\nPmPzU+iX7jo8eh6Q7iIjMfboL/Z7Yf9GiEuGxByIcXXa9tomH6tKqllaVMWSbRX81xsb+Z93t3L5\nuFw+2rKf6gYvv7x8FN88dQAV9W5Kq5v4cucB3tu4j98t2grAxSfl8L1zhzA6N+XY9pRvgr1roXyz\n9dF0ACZ9G8bfBHYH4m0kd/UfyV3+FEy4Bc66D5xxYAys/Ae8/0sYcBpc/DCkFXTpe3K8emy4dzTC\nPlESEhIOf7548WI++OADli5disvl4uyzz25z/nhs7JH/qHa7naamY38wWu5nt9vx+XwhbrnqrYwx\n7KhsYGlRFct2VFFU0UBmYgw5yXHkpcUzcUAaE/qnkRB75Ee/ttnL7FfXsXD9Ps4YmsnwPkmUHWxi\nb00z4/uncttpBYzJT8EmQnldM9v21TFncRFXzfmMJ2+ZxKCsxKPasHp3NQ++vYlVuw9yRkYtr2X/\nnqTaQjiYD0lTIOEMGH6tFd4tVe+CVc/A6uegft+R7TFJkJgNiX0gKQfyJsCAaZAz1gpWEVJcTs4Z\nns05w7O5f4qDXeuL2LpxNd4127ja2cTIAS4StwL1E+h71k/oW5DO5IJ07jpnCHtrmvD5Df1SYmDZ\nHFj0LogN7E5w18G+9eAL/qw74iF7hBXab90Nnz8OY2fCiqegbi/kngyfPgqb34Rz/wtW/h12LIbc\nCbBzCTxxCpz5IzjtbnC0+qUWYj023MMhKSmJurq6Np+rqakhLS0Nl8vFli1bWLZsWchff9q0abz8\n8svcd999vPfee1RXV4f8NVT3q6p388m2Cpq9AQLGYIwBEWxi1ZNH56Ywqm8yNtvXvwah5EAjy4sP\nYLcJTruNyno3X+w4wBc7DxyuXfdJjmVk32SqGzxs219HRZ2bgAG7TRjeJwlXjJ1cXwkXVr/AYE82\nv5s2g2sumYTN1wh71sDeNVBdDKVlsGU/nHQ1TP0ejM1l2pBM7nx+FTOe+Iz/OGMQOclxpCfE8Na6\nPbyxZg9ZSbE8c0Y1Z6z/KRKwwTk/t0a/JV/Axtfgvf+CsTfA4HOg5EsrAPets0J1yAUw5kFrxFy/\nP/hRbn2UrbC+HiA2GYZeACddA4PPg91L4fM/IUUfUgAUiI1Aen8kIROxBcAAS5+A9a/AhQ/BmOtA\nhL4p8dZo/Knvwp5V0HccOF1WoDviYPJ3rNDOPRnSBlrlFWNg6zvw4a/h44es5679OwyYCkUfw1v3\nwCu3QkwiXPYHmHg71JbBop/CRw9B7R5rezfScG8hIyODadOmcdJJJxEfH0+fPn0OPzd9+nTmzp3L\nyJEjGT58OKeeemrIX/+Xv/wlN954I88++yxTp04lJyeHpKSkkL+OCj2fP8An2yp4eUUJH24uP3wi\nsD3pCTFMHZxBuisGX8DgDwSIddhJinOQGGf9WDZ5/DR5/Izsm8yM8bmH67jvbtjHj15ZS7376L/4\n+qbEcfqQDKYMzGDq4AwKMlxH1Yzr3T5W7armi51VrC+rZWL9J8yq/j02AlzmcMPKl2FjCjTXYiUh\nEJcKKflgc8B7P4fSFTDjCU4ZlMEbd03jrhdW8ej728ighv/neJvrbLv5fh8X/VKd2Jd/BjknwQ3P\nHV2KKFsJXz5pjdBXPAU2J/Q7xRrpjptpvV5HavfCrs+sXwhb/gUbXrWOEfBCQrZ1nBGXQvogbK1H\nx2Wr4F8/hNf+A97/BcSnQ2wi7FltBfG182H01dBZrV0ERlwCwy6Cqu2QMfRITX3wOfDdpbD2JeuX\nT2p/a3tKPlz/DGz/ADKGdHz8EBBjOv5P2F0mTZpkWt+sY/PmzYwcOTIs7ekJ3G43drsdh8PB0qVL\nufPOO1mzZs1xH7e3v68dKdxfh9NuoyAzofOdgTfX7mHOx9s5Y2gmF43OoSAzgZdXlPD8st2UHWwi\nIyGGqyfkMWN8XnA+NAhWUBhjaPL6WbW7mn8XVvLFjgM0ef3YbYJdBLfPT12z7/AvBhHrBKHbF2Bw\nVgI/vHA4G8pqmLO4iHH5KfzmqjHEOe34A4aEWDt5qfFdOwFYUwpL58CyJyB/Mlz3tFWC2P4h7P4c\nkvMhb6I1Gk0InlMyxipBfPAryBwOZ/0EkvpCQibeNS9j/+IJxNeMr89YnI5gHb7vOLjg1+3XzBsP\nWCPm3PEQ07X3/xh+L+z8BLa9Z/0iGXtD5+WOgB/WPA+7loK71vpI6Q/n/woSO73BUdiJyEpjzKRO\n99Nw7zkKCwu5/vrrCQQCxMTEMGfOHCZPnnzcx+3t72tbKuvdPLJoK/+3ogSX085fvzmJ04d2POW0\n5EAj0x9bQmKcNQPD6z/yszN1UAa3TB3A+aP64Gw5U2Lfetj2LgybDjljOm2XMYZmbwARiHVYx1m0\ncR+PvLcNX0Uh/aWcs4dncfMpA4iJiYXYJIgLngCsL7dKGAd3wd511mvX7rFGjGkDrH1LvrSeB5j8\nH3DRb8ER0/U3ruhjWPAt64RiS6OuhHN/DplDu34s9bVouKvD9H21lNc2s660hpW7q3lu6S6avH6+\nOXUAS4uqKKqo5/fXj+eKcbnsOdjEwvV7AbhlagExDhv+gOHGecvYvLeWd79/JslxDj7eWsH2/XVc\nNi6XYX2SoKoImqrB22iNjlc+DSWHzs0IjLvRCsC4FKjcCpWFkJxrjZLbG7ke2AFrX8JsehOp2Nz1\nzibnQ9+xkNLPqvVWF1sj5UMnIweeaY10vw53vXW8Q7XwPqOsUbo6Iboa7l2quYvIdOCPgB140hjz\ncKvn04D5wGCgGfiWMWbDV261Uu1o8vj5ZFs5BZkJDMtO+konIz/fXsmv3trItv31gHU5+DnDs/np\npSMZnJVITZOXWc+s4O4XVzNvSREbyo5cX/H66jL+OHM8H24u58viA/zfaWXkvX4NXPEnrhg32NrJ\nGFj0M1j656NfOH0QXPgbq/67Yj588VdY/zIEWs2OErs1qp94G0y49UjtdvPb8Nos8DYiA06D6f9j\nlUoOlV78Hms2R3MtmMCRGSXJueBK/ypv71cTmxj8xfA1fzmoE6LTkbuI2IFtwAVAKbAcuNEYs6nF\nPr8D6o0xD4jICOAJY8x5HR1XR+4nTk95Xwv311FR7+a0wV/titudlQ3c+dzKw2t1JMc5GNcvlVRX\nDE67EOuwk5caR0FmAgUZCaTEO4l12vD4Avz+vW28vrqM/ukubj2tgHH5KYxOrCO+uRKyRx6uBzd7\n/fzijQ1s3lvHRaP7cNnYXLbur2P2q+to9PgxBm4ZeJCf7bsX8TWDKxO+8bI1En53Nnwx15oRMeJS\na4ZFXDL0GXP0hSsHd8OXf7NmeWSPsE7CHdxtzSAp+tA6qTfgdLj8j7D5DfjwQSvMr38GUvuF7Pug\nIlvIyjIiMhX4lTHmouDj+wGMMf/dYp9/AQ8bY/4dfFwEnGaM2d/ecTXcT5ye8L7+a91efvjKGpq9\nAa6ZkM8vrxhFctzRF8AYY9i0t5b1pTXkpMTRP93F1n11/GTBOux24cEZJ+HxBVix6wDrSmto9Pjx\n+AI0e/1UNXjafF2nXbjzrMF895wh1mXpZSvhH5eDt8Gadpc5zCqXTLunzRkS5bXN3PfqOvbu28e/\n4n6GPeCD6/5ujagbKmDwubDlbWuK4IUPdT7Loj3GWLNH3vuZNRo3ATjpWpjxZ3DGf71jqqgUyrJM\nHlDS4nEpcEqrfdYCVwP/FpEpwAAgHzgq3EVkFjALoH///l14adVTGGPYu3s7LPoZ1cOuJ3fyFaQl\ndH4iLuDz8sePivjjR0VMHJDGlIHpzFuyg6VFldxz/lBcMQ4CxlBc2ciba8soqmg45hhj81OYc9ME\n8tOsUfY1E4+dKtfo8VFc2ciuqgbq3b7Da4+cNTyLwYcusqkshOevs2aAnP8nKN8Cuz6HD35p1Y4v\n+s0x4ZydHMffb52EeXEmUrQXbn8H+k2Gb78PL1xnBftpd1uzQo5n7XwRmPBNa+rchw9C1jDruLoe\nv/qaQjXP/WHgjyKyBlgPrAb8rXcyxswD5oE1cg/Ra4dNYmIi9fX17Nmzh7vvvpsFCxYcs8/ZZ5/N\nI488wqRJ7f+ifeyxx5g1axYulxVeXVlCuDsYY9i8t45FG/fx2fZKRCAh1oHDZsNeuoyHPP9LltSQ\nVLaEGYuqcacM5tRBGVw4ug9nDM0kxm6juKqRLftq2V5eT1F5PbcU/YCZvl1kDv5Prr/tbmKdDi4a\nncMPXl7Dfa+uP+r1r86v5bHRqxjk30lt4iB2xwymLiaLs13LcSx8HCo2Wycfh1wAQ86HpCPXIbhi\nHIzKSWCUf4s1te0QTzXU97PmQD97NSDwzX9CRot6+buzrWmBvma4+H+t0f22d6wZJ00HoL4CqS2F\nSx6xgh2s1779HWvedMHpoQvhpBy48onQHEv1aiEpy7TaX4CdwFhjTLsrf0VDWeZQuHekK+F+aFXJ\n7lr9sb33tcnj550Ne9m2vx5P6Vou2DuX3zZdzQYGMb5fKnEOOw0eH2fVv8M9zXNpdOVy4KzfkvfR\n3dQ50ngw5098WFRPbbPv8LQ9ty8AWFk3M2kd/+15mMbYbFzucuh/Gkz/b8gdj8cXoLiqATEBUrYt\nIH3tPBxVW6xSSfpgazZGoMUCUxlDIWs4lC63ZmkA9DkJBp1t1b13L4ON/4SG8vbfiJhEuO1tq47d\nkjHW/O3PHgNnQrBkY7dOGiZkgSvD+qUyZZaOpFXYhbIssxwYKiIDgTJgJvCNVi+WCjQaYzzAd4Al\nHQV7TzV79mz69evHXXfdBcCvfvUrHA4HH3/8MdXV1Xi9Xh566CFmzJhx1NcVFxdz2WWXsWHDBpqa\nmrj99ttZu3YtI0aMOGptmTvvvJPly5fT1NTEtddeywMPPMDjjz/Onj17OOecc8jMzOTjjz8+Kuwf\nffRR5s+fD8B3vvMd7r33XoqLi9tdWrirlhcf4CcL1rGzsoHB9v28EvMA6eYgryVspeGaF0gdOQ28\nzfDufdbCR4PPJfna+STHp0FmIunPXsUfXH/H+/O/sry4mo+2WKE6sm8yw3OSGJIRS9zffgEyDNcd\nn8LaF61LteedBYPPJea0/2RYbAq882NrpNx3vDUyHnmFNSr2eazpgrV7rOcOjdIDAdi/3rrKb8cn\n1glKv9s6iTn0Qhh91ZErHE0AGiqtqYB1+2DkZccGO1iBff6vrK8rW2n9VTDkfIg/sX85KRVKXZrn\nLiKXAI9hTYWcb4z5jYjcAWCMmRsc3T+Ndc3yRuDbxpgOF0bpdOT+zmzrIoxQyhljrcrWjtWrV3Pv\nvffyySefADBq1CgWLVpESkoKycnJVFZWcuqpp1JYWIiIHB65twz3Rx99lA0bNjB//nzWrVvHhAkT\nWLZsGZMmTeLAgQOkp6fj9/s577zzePzxxxk7duyRMM/IgOpdFJx8BstXrGLjtiLu+n/f4csvvsAY\nwymnnMJzzz1HWloaQ4YMYcWKFYwfP57rr7+eK664gptuuunw2iHtva/NXj//++5W/v75TvJS43nk\nomxOWXwj4q63ThQu/LE1g+PS38PyJ60ZHNPutS7ptrcYCyz5nbVGxpALrJOKBdOOnh3y5d9g4Y/g\nxv+D4dOtbU0Hg1MC5x4ZfSfmWPXqsdd/vVGxt8laQTBruHWRjlJRLqTz3I0xC4GFrbbNbfH5UmDY\nV21kT3PyySdTXl7Onj17qKioIC0tjZycHL7//e+zZMkSbDYbZWVl7N+/n5ycnDaPsWTJEu6++24A\nxo4dy9ixYw8/9/LLLzNv3jx8Ph979+5l06ZNRz1Pcw00V0PAj6d8O+998BGnn38J+xoM2cmxXHnV\nVXy0+BMuvORSClotLVxYtIPC8nqavX5i7DbinHYSYx1kJB456ekPGP7zxdW8v2k/t0wdwOzJdlz/\n/JZ1ccutb1nljdvfgWevhDfusqbs3fC8NeJt7fQfWhezbHwdtr9vbcsYCmf+2DopuPi/oeAMa+2N\nQ+JT4YwfwNS7YP0CaKyESd86vlB2xkN+p//Plep1eu7CYR2MsLvTddddx4IFC9i3bx833HADzz//\nPBUVFaxcuRKn00lBQUGbS/12ZufOnTzyyCMsX76ctLQ0brvttmOP01COscXgx44j4CZbagg4vHj9\nfnZWNlBR58YW04BU7ybGLhSX15CaGE+jN0B5TSO+gCE7KQ6PL0CT109tcxONXj8muDLhg29v4v1N\n+5hzuodL6n4D8xZaq9/d+JIV7AAJmXDr29bSp2NvOHLisTWbDS54wPqoKbUWcVo6B16fZdW2PQ3t\nTw10xMLJN33l91Ap1XUn9tYgEeCGG27gpZdeYsGCBVx33XXU1NSQnZ2N0+nk448/ZteuXR1+/Zln\nnskLL7wAwIYNG1i3bh0AtbW1uBISiHUlUlxSxsJ33sHnD2CMISkpicryveBpYH8giQBCU9JAzpw2\njYUL36Zf01b6mko+e+9Nrjp1EC6bBxsB8ny7qTpwgPpmH3FOO8OyE6354RkuhvVJJCc5joONHg7U\nN7PwlScZ9uXPWZv8Qy5Zcbt1AvKs++CedTDorKM7EZ8K5/y0/WBvLSUfTr4Z7vjUGulnj4RTv2st\nCKWUCoueO3IPk9GjR1NXV0deXh59+/blpptu4vLLL2fMmDFMmjSJESNGdPj1d955J7fffjsjR45k\n5MiRTJw4EbfPT+6AQYwePpixI4eRl5vDKRPGUVVTz8Y9tcyYeQuXXz6DvD4ZPPvmRzjsNhITEyg4\nbwa33baJUy69GUyA/7jxKiaccibFVU1gj8FhtzOIvWQ7m2gKeHB4aq3ZJn4P4veS7Wsmw9HANn8F\np2/6EU1OF3GDzrUWsTrpmk7vcPOV2WxWCaetMo5S6oTShcO6gc8fwO2zPpqbm4hvLieVeozYMDYH\nIIjxgTFUx+bTZBzkeXZiXBnY2rvM3NtkBXfL5UwDfqgthaYaMMdcVgD2GHC6WL9zP19u2c5N11xD\nXJxe7ahUJAvpCVXVdfVVe3E178eFIR5BsO7CYxKysSX1sW56ANZUvwNFZLhLrIWYMEhidvsHbusS\ndJsdUgdAKtZiVD6PNf3P7rQ+xKq6OfY38+2bjr1Jt1Iqemm4h1B99X4S3ftosrmwxSZgt4HdZkNc\nGUjrGwg4YqzZJQeKrLVE4lKO756KNgfE6LdTKWXpcWlgjOna3WR6mPqDlSQ07qHJ5iK2zxBsNnvn\nX2R3WLfbqt/fbUu0hqvsppQKrx41WyYuLo6qqqqICiSPz8+BqkpcDSW4bXHEZg/uWrAfYrNb6287\n4kLeNmMMVVVVxMWF/thKqZ6tR43c8/PzKS0tpaKiItxN6ZTHF6C22YvxusmUGsrEgS0pG9vBwnA3\n7ShxcXHk53dyw2GlVNTpUeHudDoZOHBguJvRIX/A8JfF2/nDB4WcGV/MX82DSGoezm+/a10ApJRS\nPUCPCveeIhAw1l3rW9X+9x6oYc7zL1Ozt5g/5Lq5vO7/EFcfuO0tDXalVI+i4d5KIGC49e9fUlHn\n5m+3TKJfunWhz86yvdQ/eQUPmm0QA1QCmcPh5gWQ3DesbVZKqdY03Ft5dtku/l1YSYzDxlVzPuPJ\nWycTZ5ponn8lJ1HEnjMeJnfMOZCSp6sQKqV6rB41WybcSg408j/vbuGsYVksvPt04mPs3PrXxdTN\nv5qT2EbFhU+Qe96d1s2NNdiVUj2YjtyDjDHc/9p6BPjt1WPIS3Kw8MwSat//X3L8ezh40Z/oO/XG\ncDdTKaW6RMM96M1PV5K+4w2eGRNH3hf/hk1vkFRTQmKf0QTOeYSMEReHu4lKKdVlGu7AJ5tKmfzB\n9cyIqYKtWBcU5U2ESx9Fhl6APQKvmFVK9W69PtwXby1n0YuPc5a9iobL55Ewerp1ByINdKVUBOvV\n4b54azl3PLucD2Lexpc1loQJX/M+nkop1cP02tkyOysb+O7zq7gldR35gTIcZ/5Ag10pFTV65cjd\n6w9w70urcdqEH7kWQsxgGHlFuJullFIh0ytH7o9/WMja0hr+dnodMeXrYNo91uqMSikVJXpduC8v\nPsATH2/nugm5TNn9FCT1hXEzw90spZQKqV4V7v6A4UevrCU/zcVDOUtg9+dw1n3HdwckpZTqgXpV\nzX1JYQW7qhp54WInsYsfhBGXwcTbwt0spZQKuS6N3EVkuohsFZHtIjK7jedTROQtEVkrIhtF5PbQ\nN/X4LVhRSj+Xj6mrfwxJOTDjzzpDRikVlToNdxGxA08AFwOjgBtFZFSr3e4CNhljxgFnA78XkZgQ\nt/W4VDd4eH/TPuakPIPUlMA1T0J8WribpZRS3aIrI/cpwHZjzA5jjAd4CZjRah8DJIl1d4tE4ADg\nC2lLj9Mba8r4gbzAmOoP4NyfQ/9Tw90kpZTqNl0J9zygpMXj0uC2lv4MjAT2AOuBe4wxgZC0MEQC\nnz7GHY63YPJ34PTvh7s5SinVrUI1W+YiYA2QC4wH/iwiya13EpFZIrJCRFacyJtgl300l281/YOd\nOdPh4t9pnV0pFfW6Eu5lQL8Wj/OD21q6HXjNWLYDO4ERrQ9kjJlnjJlkjJmUlZX1ddv81VTvou+S\nn7IkMI60m54CW6+a/amU6qW6knTLgaEiMjB4knQm8GarfXYD5wGISB9gOLAjlA39uvxL/4IfWDT4\nZ6QmJYa7OUopdUJ0Os/dGOMTke8BiwA7MN8Ys1FE7gg+Pxd4EPiHiKwHBLjPGFPZje3umuYazKpn\neNt/KudMHh/u1iil1AnTpYuYjDELgYWtts1t8fke4MLQNi0EVj2Dw9fAC7bLeW5YZrhbo5RSJ0z0\nXqHq92KW/YUVjCZv5KnEOnRhMKVU7xG9Zxc3vYHUlvEXz8VcPKZvuFujlFInVHSGuzGw9M9UxPbn\nC8dEzhp2gmbmKKVUDxGd4V65DfasZr7nAs4ZkUOcU0sySqneJTrDffdSAN5pHsUlWpJRSvVC0XlC\ndfcy6h2p7PPncvZwLckopXqfqBy5m91LWe4fzjnD++CKic7fX0op1ZHoC/e6fUh1MZ96hnDBqD7h\nbo1SSoVF9IX77mUArAgMZ9KA9DA3RimlwiMqw90jseyLH0a/9Phwt0YppcIi+grSu5eyyTaUMf0z\nEV3aVykP0jdQAAAPTUlEQVTVS0XXyN1dj9m3niXuIZzcX2+hp5TqvaIr3MtWIMbPisBwTu6XGu7W\nKKVU2ERXuO9ehkFYY4YyVsNdKdWLRVfNffdSSpwDyU3tQ2JsdHVNKaW+iugZuft9mNIVfO4dyngd\ntSulernoCfeKzYinns89Qzi5v4a7Uqp3i6Jw3wrAVtNPZ8oopXq96An3qu0EEKpi8hmSpTfCVkr1\nbtFz1rFyG+W2bEbkZmOz6cVLSqneLWpG7oGKQrb4+mq9XSmliJZwDwQwVYUUBfoyJi8l3K1RSqmw\ni45wry3D7muiyOQyMDMh3K1RSqmwi45wryoEoCiQS36aK8yNUUqp8IuOcK+0wr02YQDxMXozbKWU\nipJw30aDJJCQnhvuliilVI/QpXAXkekislVEtovI7Dae/7GIrAl+bBARv4icuNsgVRZSTB79M7Te\nrpRS0IVwFxE78ARwMTAKuFFERrXcxxjzO2PMeGPMeOB+4BNjzIHuaHBbTGUhW3x96Jeu9XallIKu\njdynANuNMTuMMR7gJWBGB/vfCLwYisZ1ibsOqdtDUSCX/hruSikFdC3c84CSFo9Lg9uOISIuYDrw\n6vE3rYuqtgNQZHLpn6HhrpRSEPoTqpcDn7VXkhGRWSKyQkRWVFRUhOYVgzNlikxfHbkrpVRQV8K9\nDOjX4nF+cFtbZtJBScYYM88YM8kYMykrK6vrrexI5TYC2Nln70tWYmxojqmUUhGuK+G+HBgqIgNF\nJAYrwN9svZOIpABnAW+EtomdqCykwtmXnPQUXTBMKaWCOl0V0hjjE5HvAYsAOzDfGLNRRO4IPj83\nuOtVwHvGmIZua21bKgspRksySinVUpeW/DXGLAQWtto2t9XjfwD/CFXDuiTgx1RtZ5PvQg13pZRq\nIbKvUK0pQfxutvhydI67Ukq1ENnhXl0MwC7TR0fuSinVQmSHe3MtAAdNooa7Ukq1ENnh7q4DoJ54\n+qXHh7kxSinVc0RFuMe4UnDFRM/tYJVS6nhFRbhnpJ+4BSiVUioSRHi419JMDHkZyeFuiVJK9SgR\nHe7+5lrqTLyeTFVKqVYiOtyb6g5SZ+J1jrtSSrUS0eHuaayhnnjyUnWmjFJKtRTR4W7z1FFv4nHF\n6kwZpZRqKcLDvZ564ol1RHQ3lFIq5CI6Fe3eeuo03JVS6hgRnYoObz31Jp5Ypz3cTVFKqR4lcsPd\nGBw+LcsopVRbIjcVfW7sxke9cWm4K6VUK5GbisGlB6yau5ZllFKqpQgOd2u53wbicdr13qlKKdVS\nBIe7NXJvtrkQ0XBXSqmWIj7cPfbEMDdEKaV6nggOd6ss43YkhLkhSinV80RwuFsjd69dw10ppVqL\n+HD36chdKaWOEcHhbpVlfM6kMDdEKaV6nggO9zp8OLA5Y8PdEqWU6nEiOtwbxaXryiilVBu6FO4i\nMl1EtorIdhGZ3c4+Z4vIGhHZKCKfhLaZbXDX0Sh6dapSSrWl07tciIgdeAK4ACgFlovIm8aYTS32\nSQXmANONMbtFJLu7GnyYu44GdF0ZpZRqS1eScQqw3RizwxjjAV4CZrTa5xvAa8aY3QDGmPLQNrMN\n7joa0OV+lVKqLV0J9zygpMXj0uC2loYBaSKyWERWisgtbR1IRGaJyAoRWVFRUfH1WnyIu1Zv1KGU\nUu0IVTI6gInApcBFwH+JyLDWOxlj5hljJhljJmVlZR3fK7rrqDMa7kop1Zau3Fm6DOjX4nF+cFtL\npUCVMaYBaBCRJcA4YFtIWtkWdx21gTg9oaqUUm3oyrB3OTBURAaKSAwwE3iz1T5vAKeLiENEXMAp\nwObQNrUVdx01gXhidOSulFLH6HTkbozxicj3gEWAHZhvjNkoIncEn59rjNksIu8C64AA8KQxZkO3\ntdrnAV8zNYE4LcsopVQbulKWwRizEFjYatvcVo9/B/wudE3rgKcegHriyXZquCulVGuRmYzBdWXq\n9RZ7SinVpggN9+D9U3W2jFJKtSkykzEY7vU6z10ppdoUmcl4KNyNXqGqlFJtiexw15G7Ukq1KTKT\nMXhCtc7owmFKKdWWyEzGo0buWpZRSqnWIjbcDUIjscTqPHellDpGZCajuw6fMxEQLcsopVQbIjMZ\n3XV4HYkAGu5KKdWGyExGdy1eRwKA1tyVUqoNERrudXjsh8I9MruglFLdKTKT0V2H264jd6WUak/E\nhnuzzQWgs2WUUqoNkZmM7jqaxBq5x9gjswtKKdWdIjMZ3XU02VzE2G3YbBLu1iilVI8TeeEe8IOn\nnibRpQeUUqo9kZeOwbswNUi81tuVUqodkZeOh9eVcelMGaWUakfkhrvehUkppdoVeel4+BZ7ccRo\nuCulVJsiLx2Da7nX6shdKaXaFXnpGBy51wTitOaulFLtiLxw73cKXP8sZSZLZ8sopVQ7Ii8dk3Nh\n1BXU+GO1LKOUUu3oUjqKyHQR2Soi20VkdhvPny0iNSKyJvjxi9A39Whun1/LMkop1Q5HZzuIiB14\nArgAKAWWi8ibxphNrXb9tzHmsm5oY5vcvoCO3JVSqh1dSccpwHZjzA5jjAd4CZjRvc3qnNsX0Jq7\nUkq1oyvpmAeUtHhcGtzW2mkisk5E3hGR0SFpXQc8voCWZZRSqh2dlmW6aBXQ3xhTLyKXAP8Ehrbe\nSURmAbMA+vfvf1wvaNXcdeSulFJt6Uo6lgH9WjzOD247zBhTa4ypD36+EHCKSGbrAxlj5hljJhlj\nJmVlZX3tRhtjtOaulFId6Eo6LgeGishAEYkBZgJvttxBRHJERIKfTwketyrUjT3E6zcYA7FOLcso\npVRbOi3LGGN8IvI9YBFgB+YbYzaKyB3B5+cC1wJ3iogPaAJmGmNMdzXa7fMDenNspZRqT5dq7sFS\ny8JW2+a2+PzPwJ9D27T2uX0BAF04TCml2hGR6Xgo3HXkrpRSbYvIdHR7D5VltOaulFJticxw15G7\nUkp1KCLT8XC46xWqSinVpohMRy3LKKVUxyIy3D1+LcsopVRHIjId3d5D4a4jd6WUaktkhrvW3JVS\nqkMRmY56hapSSnUsItPxyFRILcsopVRbIjPcvTpyV0qpjkRkOuraMkop1bGITEe9QlUppToWkeno\n9vmx2wSHPSKbr5RS3S4i09Ht1bswKaVURyIyIfUWe0op1bGITEjr5tg6DVIppdoTkeHu8QX06lSl\nlOpARCaklmWUUqpjEZmQVrhrWUYppdoToeHu15G7Ukp1ICIT0u3VmrtSSnUkIhPS7QsQoxcwKaVU\nuyIyIXUqpFJKdSxCw13LMkop1ZGITEhdfkAppTrWpYQUkekislVEtovI7A72mywiPhG5NnRNPJaW\nZZRSqmOdhruI2IEngIuBUcCNIjKqnf3+B3gv1I1sTS9iUkqpjnUlIacA240xO4wxHuAlYEYb+/0n\n8CpQHsL2tUmXH1BKqY51JSHzgJIWj0uD2w4TkTzgKuAvHR1IRGaJyAoRWVFRUfFV2wqAzx/AFzBa\nllFKqQ6Eavj7GHCfMSbQ0U7GmHnGmEnGmElZWVlf64U8fr0Lk1JKdcbRhX3KgH4tHucHt7U0CXhJ\nRAAygUtExGeM+WdIWtmC26vhrpRSnelKuC8HhorIQKxQnwl8o+UOxpiBhz4XkX8Ab3dHsEOL+6c6\ntSyjlFLt6TTcjTE+EfkesAiwA/ONMRtF5I7g83O7uY1Hcfv8gI7clVKqI10ZuWOMWQgsbLWtzVA3\nxtx2/M1q36GRe4yGu1JKtSviEvJIzV3LMkop1Z7IC3ctyyilVKciLiEPn1DVcFdKqXZFXEIeHrnr\nbBmllGpXxIW7R0fuSinVqYhLyKykWC4Zk0OqyxnupiilVI/VpamQPcnEAelMHJAe7mYopVSPFnEj\nd6WUUp3TcFdKqSik4a6UUlFIw10ppaKQhrtSSkUhDXellIpCGu5KKRWFNNyVUioKiTEmPC8sUgHs\n+ppfnglUhrA5kaI39rs39hl6Z797Y5/hq/d7gDGm05tQhy3cj4eIrDDGTAp3O0603tjv3thn6J39\n7o19hu7rt5ZllFIqCmm4K6VUFIrUcJ8X7gaESW/sd2/sM/TOfvfGPkM39Tsia+5KKaU6Fqkjd6WU\nUh2IuHAXkekislVEtovI7HC3pzuISD8R+VhENonIRhG5J7g9XUTeF5HC4L9p4W5rqImIXURWi8jb\nwce9oc+pIrJARLaIyGYRmdpL+v394P/vDSLyoojERVu/RWS+iJSLyIYW29rto4jcH8y2rSJy0fG8\ndkSFu4jYgSeAi4FRwI0iMiq8reoWPuCHxphRwKnAXcF+zgY+NMYMBT4MPo429wCbWzzuDX3+I/Cu\nMWYEMA6r/1HdbxHJA+4GJhljTgLswEyir9//AKa32tZmH4M/4zOB0cGvmRPMvK8losIdmAJsN8bs\nMMZ4gJeAGWFuU8gZY/YaY1YFP6/D+mHPw+rr08HdngauDE8Lu4eI5AOXAk+22BztfU4BzgSeAjDG\neIwxB4nyfgc5gHgRcQAuYA9R1m9jzBLgQKvN7fVxBvCSMcZtjNkJbMfKvK8l0sI9Dyhp8bg0uC1q\niUgBcDLwBdDHGLM3+NQ+oE+YmtVdHgN+AgRabIv2Pg8EKoC/B8tRT4pIAlHeb2NMGfAIsBvYC9QY\nY94jyvsd1F4fQ5pvkRbuvYqIJAKvAvcaY2pbPmesaU5RM9VJRC4Dyo0xK9vbJ9r6HOQAJgB/Mcac\nDDTQqhQRjf0O1plnYP1yywUSROTmlvtEY79b684+Rlq4lwH9WjzOD26LOiLixAr2540xrwU37xeR\nvsHn+wLl4WpfN5gGXCEixVjltnNF5Dmiu89gjc5KjTFfBB8vwAr7aO/3+cBOY0yFMcYLvAacRvT3\nG9rvY0jzLdLCfTkwVEQGikgM1smHN8PcppATEcGqwW42xjza4qk3gVuDn98KvHGi29ZdjDH3G2Py\njTEFWN/Xj4wxNxPFfQYwxuwDSkRkeHDTecAmorzfWOWYU0XEFfz/fh7WuaVo7ze038c3gZkiEisi\nA4GhwJdf+1WMMRH1AVwCbAOKgJ+Fuz3d1MfTsf5UWwesCX5cAmRgnV0vBD4A0sPd1m7q/9nA28HP\no77PwHhgRfD7/U8grZf0+wFgC7ABeBaIjbZ+Ay9inVPwYv2V9u2O+gj8LJhtW4GLj+e19QpVpZSK\nQpFWllFKKdUFGu5KKRWFNNyVUioKabgrpVQU0nBXSqkopOGulFJRSMNdKaWikIa7UkpFof8PoqSi\nixb53ssAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1f06c53e780>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['training', 'validation'], loc = 'upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training and validation accuracy improve instantaneously, but reach plateau after around 50 epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 9472/10000 [===========================>..] - ETA: 0s"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy:  0.9248\n"
     ]
    }
   ],
   "source": [
    "print('Test accuracy: ', results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Batch Normalization\n",
    "- Batch Normalization, one of the methods to prevent the \"internal covariance shift\" problem, has proven to be highly effective\n",
    "- Normalize each mini-batch before nonlinearity\n",
    "- Doc: https://keras.io/optimizers/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/torch/torch.github.io/master/blog/_posts/images/resnets_modelvariants.png\" style=\"width: 500px\"/>\n",
    "\n",
    "<br> Batch normalization layer is usually inserted after dense/convolution and before nonlinearity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.layers import BatchNormalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def mlp_model():\n",
    "    model = Sequential()\n",
    "    \n",
    "    model.add(Dense(50, input_shape = (784, )))\n",
    "    model.add(BatchNormalization())                    # Add Batchnorm layer before Activation\n",
    "    model.add(Activation('sigmoid'))    \n",
    "    model.add(Dense(50))\n",
    "    model.add(BatchNormalization())                    # Add Batchnorm layer before Activation\n",
    "    model.add(Activation('sigmoid'))    \n",
    "    model.add(Dense(50))\n",
    "    model.add(BatchNormalization())                    # Add Batchnorm layer before Activation\n",
    "    model.add(Activation('sigmoid'))    \n",
    "    model.add(Dense(50))\n",
    "    model.add(BatchNormalization())                    # Add Batchnorm layer before Activation\n",
    "    model.add(Activation('sigmoid'))    \n",
    "    model.add(Dense(10))\n",
    "    model.add(Activation('softmax'))\n",
    "    \n",
    "    sgd = optimizers.SGD(lr = 0.001)\n",
    "    model.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = ['accuracy'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = mlp_model()\n",
    "history = model.fit(X_train, y_train, validation_split = 0.3, epochs = 100, verbose = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmUXWWd//v394xVp+YplaQqSVUGyAAhkBjCKKOGySAy\nBEQR7U6D+EN79f1d0b5Xm+52Nf2Tnwv9tUijol4FI4JI1ABOQVAGk0CICUlIZa6MNQ9nqDN97x/7\nJDmp1ERyKifn1Pe1Vq3U2XvX2c9TlXzqyXfv/TyiqhhjjMkvrmw3wBhjTOZZuBtjTB6ycDfGmDxk\n4W6MMXnIwt0YY/KQhbsxxuQhC3djjMlDFu7GGJOHLNyNMSYPebJ14urqam1oaMjW6Y0xJietXbu2\nVVVrhjsua+He0NDAmjVrsnV6Y4zJSSKyayTHWVnGGGPykIW7McbkIQt3Y4zJQ1mruQ8kFovR3NxM\nJBLJdlPyRkFBAfX19Xi93mw3xRhzCp1W4d7c3ExJSQkNDQ2ISLabk/NUlba2Npqbm2lsbMx2c4wx\np9BpVZaJRCJUVVVZsGeIiFBVVWX/EzJmDDqtwh2wYM8w+34aMzadVmUZY4zJNdF4kh2tQbYc7GFP\ne4iyQi8TygqoKfHTFY6xvytCS08f9RWFzJ9SQV154SkZdFm4p+ns7OSpp57is5/97Pv6umuvvZan\nnnqK8vLyQY/5yle+wqWXXspVV111ss00xmSKKiSikIwf3ZaIQSxEd1cnu/YfpEiDBDSIT6MkPMVE\nvcV0xT1s2bWfpuZ9HGxpI5aEJC4UoVx6aZNOqugmiYs+vICHNqK8ISGqPRFcs67n0ls/P6pdG1G4\ni8hi4JuAG/ieqj7Ub38F8AQwDYgAn1bVDRlu66jr7Ozk0UcfPS7c4/E4Hs/g36qVK1cO+97/+q//\netLtMybvJJPQ1wWhdujrBnGDywMoBFuh9xCEO8AXAH8p+IshFoZIN0S6INwOoTbnmGQCdXkIxiAS\n6iUZ6UL6uvHGQ/iSIbyJMCTjKM7NBi5N4NXooE0rBc4eZF8dMPvwiwGiQcVFzFuGiOBO9iGJPpLu\nQsLuYrqShbS6R/862LDhLiJu4NvA1UAzsFpEVqjqu2mHfRlYp6ofFZGZqeOvHI0Gj6YHHniAbdu2\nMW/ePLxeLwUFBVRUVLB582bee+89brzxRvbs2UMkEuHzn/88y5YtA45OpdDb28s111zDxRdfzGuv\nvUZdXR3PP/88hYWFfOpTn+L666/n5ptvpqGhgbvuuotf/epXxGIxfv7znzNz5kxaWlq444472Ldv\nHxdccAG/+93vWLt2LdXV1Vn+zpi8pgrRXicsI90QD0M8CvEIxPucPxNREBd4CsDjh74eCLY44QtQ\nUAoFZc6oN7Vd+3qIx2PEolE0GsLT14G7rwPp63FCNhnHlYjiInnCTU8ihFwlRH1l9CVdRKNRXMk4\nYXz0EKBHA/RSS0j9hPATS0Wez+MilhSCSS996iWedvkxgQtvYSmzpkxgWl0tEXcxPQQIJT34k2EK\nk70EJMq0uvGUllc5v3BUIZkATUCgCglU4XO5j2mrGyhOfdSdcI9HbiQj94VAk6puBxCR5cASID3c\nZwMPAajqZhFpEJFaVT14og178FcbeXdf94l++YBmTyzlqzfMGXT/Qw89xIYNG1i3bh0vv/wy1113\nHRs2bDhyG+ETTzxBZWUl4XCYD3zgA3zsYx+jqqrqmPfYunUrP/3pT/nud7/LrbfeyrPPPsudd955\n3Lmqq6t56623ePTRR3n44Yf53ve+x4MPPsgVV1zBl770JV588UW+//3vZ7T/JgepOn8OVKNNJiHS\n6Yx6Q6lRbvCQ8zraC9EQxEJHQzoWdl4f3hcNOh+xIOiJBWwiFYrutIBOInRQSrcWElM3Cdz04aVD\ni2lnCr1aSBwPcVxE8dKpxfS4SwlJMdFYDDdJBKWdUpKBGkorapF4iHi4C1e0h8JACWUV1ZRVVLGj\n18vmQyF2tgWpLSng/JmVnN9YxbSaIqqK/Uwt8iECvX1xgn0JfB4XtaV+Aj4PiaSyuz3ElgPdtAWj\nFHrdFHrdjC8r4Jz6clyu3L4ZYSThXgfsSXvdDJzf75h3gJuAV0VkITAFqAeOCXcRWQYsA5g8efIJ\nNvnUWbhw4TH3h3/rW9/iueeeA2DPnj1s3br1uHBvbGxk3rx5AMyfP5+dO3cO+N433XTTkWN+8Ytf\nAPDnP//5yPsvXryYioqKjPbHZEm4Ezp3Qece6Gp2RrbhdieE431OvVcT4PaBrwi8AafM0L4DOnY6\no+aCUhK+UmLqwhMP4oqHkWgPMkgoq9uP+ALgLUI9foIJD10xF32uQmLuMqKuWoIePz0uP70+P+5A\nOSXllRQUV7CpNc6be4K0R4QIPqJ4wO0jnkjgJ8aEImF/2M2BRCkTJ0zk7PpyfMkIBYlexOUi4qsA\nl5dCn5vKgI+KIh+FXjexRJJEIkmRCLWlfsaXOhcdi/0ePG7nl0QoGudgdx/d4RhTqgKUB3wj+hZH\n40m8bhn0QuVA7+N2CY3VRTRWF43s55hjMnVB9SHgmyKyDvgb8DaQ6H+Qqj4OPA6wYMECHeoNhxph\nnypFRUd/6C+//DK///3vef311wkEAlx22WUD3j/u9/uPfO52uwmHwwO+9+Hj3G438Xh8wGPMKRaP\nQs9+aNsKrU3OKDgZd/67nYgdHQGj4C9xyhDewNF9iRi4XE7NOBmHli1wcCN07z32POKCwkoorABv\nAbg8JHARDkeIRXqQaJCop4SC2mmUnncpUZePd7c3s3f/flSTBHUSIfwEJUDCX4GrqIqgp4x1HX62\nhQJ0UEJlSRGXTq1hfGkBz729l72dYQq9bvxeF8mk80+vpMBLWaGXwkI3+zrD7N8dSW33cNWsWm6c\nXcvMCaWMK/ET8LnZ1Rbiz02tvLG9jYvKCvjY/Hpmji/N6I8g4PPQWP3+Y8nnOe3u6s66kXwX9wKT\n0l7Xp7YdoardwN0A4vzq3AFsz1AbT5mSkhJ6enoG3NfV1UVFRQWBQIDNmzfzxhtvZPz8F110EU8/\n/TRf/OIX+e1vf0tHR0fGz5EXkkmI9kAingrfmFNyiPY6ZYaeA05Id+939rs8Tuj29UBvixPafT2p\nenJfqlQRdN4nnbjB7U19vRs8BSTdfhIKnlgvEulyRtwAbj+4vagmSSZiJBXa/JPpKj2HWP1S4mUN\nJMomkSieyP5YgObOPpo7wuxsDbK9tZeD3X3OKQWm1RSzpz1E33tJzgqWsr8zQlswyg3nTOQj50wk\n3hcnEo4R6eljb2eY5o4Q0XiSM2eX8OEJpRR63fy5qZXfvXuQrnCMi6dX8+VrZ3H17NohQzDYF2d/\nV5jJlUUDHtdQXURDdRF3LpqSsR+lGT0jCffVwAwRacQJ9aXAHekHiEg5EFLVKPB3wCupwM8pVVVV\nXHTRRZx11lkUFhZSW1t7ZN/ixYt57LHHmDVrFmeeeSaLFi3K+Pm/+tWvcvvtt/PjH/+YCy64gPHj\nx1NSUpLx85yWokFoa4LWrdC9D0j9xy4ehe5mp6TRvde5MyLUfjRUh+INOKWOZMIJbl8xFNdCcQ2U\nTnQuDrp94C1MlUOKoKgaqmdA1QwoHnek1r2jNchTb+7i52ub6QzFmFBWwBVn13BhQylxcROMKrva\ngjyztpm2UJSaEj/xaJKO3THYfbhBIaDpSPMqi3w0VAW4eHoNU2uKmDOxlHMnV1BW6KUrFOOX6/by\nzNpmZk8s5Z8+dCbzJg1+q21/SxdOJpFUusMxKopGVtoo8nuYPm6M/H0bA0R1yOqIc5DItcAjOBd8\nn1DVr4nIPQCq+piIXAD8COdf5EbgM6o65LBzwYIF2n+xjk2bNjFr1qwT6kg+6Ovrw+124/F4eP31\n17n33ntZt27dSb/vKf++JuLObW2Hyxjd+6BlExzanCp1JJwRdTR49CJgeIi/LoFqKJ8EpXVQVAOB\nVEnD7XdG1C6PE+S+IueWueJaJ7z9pUfCuelQL6FonKpiP5UBH1sP9bBqcwurthwi2BdncmWAyVUB\n6isCjC8tYHyZn96+BK81tfLatjb+trcLj0v48JzxnD+1kr80tfLq1lZC0aO/ZFwCV86q5c5FU7hk\nejUulxCKxjnQFSEUTdAXTxKNJ6ku9lFXUUjAZ4+ZmPdPRNaq6oLhjhvR3y5VXQms7LftsbTPXwfO\neL+NNMfavXs3t956K8lkEp/Px3e/+91sN2l48ahz0a9tK+x9C/a8CXvXOqWO/nzFTkC7vakyRyHU\nnAGNl0DJeGe0XD0DyiY5+8EJbo//+PdKiSWS7OsMU1XsXJhL1xOJ8at39vOz1bt5p7nruK8VgXPq\ny2msLmJ3e4jXt7cdE9YAXrdw7uQK/ueHz+SW+fWMKy0A4JMXNBCJJdjeEsTvdVHk81Ba6DkusAM+\nD1NrikfwjTQms2zocBqZMWMGb7/9drabcaxE3CmHdO91RuDd+5w7P9p3QMcO6Nh1tEQibhh/Npz3\nSahoSN0TXeCUOmpmQln9wLf0pagq7cEorZ1RygNuygPONMUbdrWzdlcHm/b3EE9dDIzFk2xv7WVH\na5BYwtk2vrSAhuoAoWiCfZ1hWnudB1TOqC3m/71+NpMrA7QH+2jtjTK+tIDLzqyhqth/zPk7QzEO\ndEc40B3B4xLmT6kYdIRd4HUze2JmLygakykW7sYpk7RuhYMbnFF41x6nxt2xAzp3H/toNoC/DCob\nYcI5MOemVI16Ooyb5ZRGBjpFUtlyoIedrUEi8QR9sSTdkRgHu/s40B1hX2eY7S1BusLHXtR0CaTy\nnIllBRR4nRG9yyU0VAW4clYtjVVFtPT2sa2ll52tQcoDPuZMLKWuvJALp1dz7qTyEc3lISJUFDm3\n7s2aYKFtcpuFe75TdW7Re/snzuPc4Xanvh0NOg+yhNqc2/ViwaNfE6hySiPj58LsJVDR6Iy6Syc6\n5ZOC8iFH4IftaQ/x6tZW/tLUyuvb22gPHv+o9+GHRiaUFXD93AlMrSlmXImf7kiMjmCUvniSORPL\nmD+lgpqSwcszxphjWbjnm2TcuYiZiDrhHelybgt86T5nv9vnXIz0FTsXHwvK4bxPwIR5TkmlsnHQ\n0fdAmjtCrN7ZzuqdHXSGnPBWhS0Hetje6vzCOFwCuXBaNbMnlFLoc+P3uCgu8FDi99i0xMaMAgv3\nXKdJZxR+eCKlRF/aTnEetimshP/xlnMXia9owFF3OJrg5S2HOC/gprbfnXNd4Rhvbm/j9e1trN7Z\nTmcoRjSeJBJL0B1xSjYlBR7Gpy42AkypCnDnoilcekY102qKLcCNOcUs3E9CcXExvb297Nu3j/vv\nv59nnnnmuGMuu+wyHn74YRYsGPzOpUceeYRly5YRCASAYaYQjkWc0ko06IzOE4dLHeJMYBSocu4u\n8fhTtwq64FAUqqYdeQtVPSZs3zvYw31PvsXWQ714XMKH5tRy47w6mlp6WbX5EGt3dZBU8HtczJ9S\nwRnjSvC6Xfg8LqbVFLGwsYozx5fgzvG5OIzJJxbuGTBx4sQBg32kHnnkEe68884j4X5kCuHDM/Ml\n407dvK/LCXVw7uv2FqUeYQ84I/R+s9ABJFUJRxP8Xz9/hx2tQXa1BekOxzl/aiUfPKMGr9vFf7yw\niWK/h28unceGvV38fG0zK/92AIA5E0u57/LpXDy9mnmTy/F7jj+HMeb0Y+Ge5oEHHmDSpEncd59T\nn/6Xf/kXPB4Pq1atoqOjg1gsxr//+7+zZMmSY75u586dXH/99WzYsIFwOMzdd9/NO++8w8yZM4+Z\nW+bee+9l9erVhMNhbr75Zh588EG+9a1vsW/fPi6//HKqq6tZtWoVDQ1TWPO7Z6kucvON//4JT/zs\neQD+7uM384Uv/CM7D3VxzZVLBpxaGJyReSSWoCscpyMUpS0YZdXmNmbUFnPVrFoKUo+n//tvNgFw\n4bQqHlk6j3ElBSyZV8c/fehMVu9s54zaEmrTSi3GmNxx+ob7Cw/Agb9l9j3Hnw3XPDTo7ttuu40v\nfOELR8L96aef5qWXXuL++++ntLSU1tZWFi1axEc+8pFBa8jf+c53CAQCbNq0ifXr13Peeecd2fe1\nr32NyspKEokEV155JevXr+f+++/nG9/4BqtWPkd1aaEz2VQiBn1B1u4L84NnX+TNN95EXW7Ov+Ai\nPnj9bVRUVBwztfAtt97Kk8uf5vqbbiXYlyAcS5DUo5NDVRf7+Os/X3Vc2WRPe4hdbSEumFZ1zL4C\nr5tLZtSc8LfZGJN9NpVamnPPPZdDhw6xb98+3nnnHSoqKhg/fjxf/vKXmTt3LldddRV79+7l4MHB\np6l/5ZVXjszfPnfuXObOnXtk39NPP815553Hueeey8aNG3n3nbXQts2pm3ftdqaCRZynMsfN5M/r\ntvDRmz5GUXkVxaXl3HTTTbzyyit0hqJMmtJAxaQZNB3qpX76HNa9u5WWniiKM2fJ5MoAM8eX0lhd\nRIHXPWA9fFJlgItnVFut3Jg8dPqO3IcYYY+mW265hWeeeYYDBw5w22238eSTT9LS0sLatWvxer00\nNDQMONXvcHbs2MHDDz/M6jffoKIAPvWZvyfSutt5TN/lhsrpMKHemQ5WXMfVz6PxBL2RGPHuCPu7\nIni8PiIxZw7r4gIfMYkze2IJbpf9vjbG2Mj9OLfddhvLly/nmWee4ZZbbqGrq4tx48bh9XpZtWoV\nu3btGvLrL730Up566ikANmzYwPr16wHo7uygqNBHWaSZg03reOEPrzoTYI2bQ0lpOT19CRAX8YSz\n+EIimeSiiy7m2eee493dLby1/QC/WrGCiy++hMmVAfweF2eOL2FqTTFlAW9qdG4/TmOM4/QduWfJ\nnDlz6Onpoa6ujgkTJvDxj3+cG264gbPPPpsFCxYwc+bMIb/+3nvv5e6772bWrFnMmjWL+fPnQyzC\nOZMqOXfWdGZ+8KNMmjSFiy65NHWHi4tly5bx4cWLqR43nv9evoJYIsnmAz1UjJ/GNR9dypIPfRC3\nS7hn2d9z3eUXDrq6kzHGHDaiKX9Hw5iY8lfVWduya69TR69ocO5FB+KJJJHUFLDhWIKOYBRVqChy\nlidLJp3bGAu9bkoKTu4pzrz7vhozhmV0yl/zPqk6Cxf3HHRWkveXQPkUZ6pboD3Yx96OCJpakEIQ\nygNexpX67T5yY0xGWLhnUiLmTMoVanMePnL7oXyy8/i/CKrKwZ4+DnVHjtyi6PO48LpduOzxfGNM\nBp124d7/0fjTnqqz6lDvIWcNT3CWbSuf4jw9mupLXzzBoe4+OkJRKgLOSjynItCzVXYzxmTXaRXu\nBQUFtLW1UVVVlRsB39cLPfucKQHcPigef3RFeyASS9AejNITidMXdxa0GFdSQG2p/5T0T1Vpa2uj\noMCeMjVmrBlRuIvIYuCbOGuofk9VH+q3vwz4CTA59Z4Pq+oP3m9j6uvraW5upqWl5f1+6amVjEO4\n8+g96v4y8PlBOoFOovEkPZEY4VgSEWfCrQKvmwKPi44eFx37Tl1TCwoKqK+vP3UnNMacFoYNdxFx\nA98GrgaagdUiskJV30077D7gXVW9QURqgC0i8qSqHr86wxC8Xi+NjY3v50tOrb5eePV/w+v/BS4v\nXPKPsOizxN2FvLq1lVVbDvHylhZ2t4coD3j51IUN3HVBw4hXnzfGmEwZych9IdCkqtsBRGQ5sARI\nD3cFSsSpNRQD7UC8/xvltOY18OxnnGXo5i6Fq74KpRPpiye47ydr+f2mQxR63Vw4rYp/+OBUbpxX\nR5H/tKp6GWPGkJGkTx2wJ+11M3B+v2P+C1gB7ANKgNtUNZmRFmZbMgGvfgNe/g8orYO7X4ApFwJO\nTf0ffryWP73Xwv9z3SzuXDTlyBqfxhiTTZkaWn4YWAdcAUwDficir6pqd/pBIrIMWAYwefLkDJ16\nFPX1wM8+AdtXwVk3w3X/GwqdBTR6IjHu/clb/GVbKw/ddDZLF+ZAf4wxY8ZIwn0vMCntdX1qW7q7\ngYfUue+uSUR2ADOBv6YfpKqPA4+D84TqiTb6lOg5CE/e7CwefcO3YP5ddIViPP/6Tn6/6RBvbGsj\nlkzy9ZvP4eb5dsHSGHN6GUm4rwZmiEgjTqgvBe7od8xu4ErgVRGpBc4EtmeyoadU2zb48UedKXjv\n+BnMuJo97SE+8f032dkWoqEqwCcumMIN50xk3qQBlsIzxpgsGzbcVTUuIp8DXsK5FfIJVd0oIvek\n9j8G/BvwQxH5GyDAF1W1dRTbPXpat8IPr3Nud7zr11A/n/cO9vCJ779JOJpg+bJFLJpale1WGmPM\nkEZUc1fVlcDKftseS/t8H/ChzDYtC9q2wY9ucC6ifmoljJvJO3s6uesHf8XndvH0PRcwc3xptltp\njDHDsnv1Dmvf4QR7IuqM2MfNpDMUZdmP11Ds9/DU3y1iclUg2600xpgRsXAH52nTn9zkPHF616+g\ndjaqyj8/t4G23ii/vO8iC3ZjTE6xpXtU4Zefhc7dcPtyZxFt4Lm39/Kbv+3nH68+g7PqyrLcSGOM\neX9s5P6Xb8KW38Dih2DyIgD2tIf46vMb+UBDBfd8cFqWG2iMMe/f2A73Ha/CHx6EOTfB+fdwqCfC\nz/66h5+8uQsFvnHrPNyuHJid0hhj+hm74R5shWc+DVUz0I98i/94YTM/+MsOYgnlkhnVfP7KGUyq\ntDq7MSY3jc1wV4Vffd5ZCu+Tv+Tn6zt5/JXt3HRuHZ+7YjpTa4qz3UJjjDkpYzPc3/4xbP41fOhr\nvMdkvrLiz1w4rYqv33KOlWGMMXlh7N0t074dXngAGi8lPP8fuO/Jtyj2e3jkNquvG2Pyx9gauavC\nc/eC2wM3fod//c1mmlp6+f8+vZBxpbYUnTEmf4ytkft7L8KeN+Dqf6Opr5zlq3fz6YsauWRGTbZb\nZowxGTV2wl0V/vS/oHwKzLuD//7TNvweF/deZvexG2Pyz9gJ96Y/wL634JJ/Ym9PnOfe3svSD0ym\nutif7ZYZY0zGjY1wV4U//SeUTYJzbue7rzhTzf/9pVOz3DBjjBkdYyPcd/wJmv8KF3+BtoiyfPVu\nlsyro668MNstM8aYUTE2wv1P/wtKJsK5n+CHr+2kL57k3sts1G6MyV/5H+6HNsOuv8AF9xFRDz96\nbScfnj2e6eNKst0yY4wZNfkf7hueBXHB2bfwwob9dEfi3HVhQ7ZbZYwxoyq/w10VNv4CGi6GklqW\n/3UPU6oCLJpame2WGWPMqBpRuIvIYhHZIiJNIvLAAPv/p4isS31sEJGEiGQ/QQ+sh7YmOOtj7GwN\n8uaOdm5dMAkRm2bAGJPfhg13EXED3wauAWYDt4vI7PRjVPXrqjpPVecBXwL+pKrto9Hg92XDs+Dy\nwKyP8PSaPbgEbp5fn+1WGWPMqBvJyH0h0KSq21U1CiwHlgxx/O3ATzPRuJOiChueg2lXEPeX88za\nZq6YOY5am0PGGDMGjCTc64A9aa+bU9uOIyIBYDHw7Mk37SQ1r4Gu3TDnJl7e0sKhnj5uXTAp260y\nxphTItMXVG8A/jJYSUZElonIGhFZ09LSkuFT97PhWXD7Yea1LF+9h5oSP5fPHDe65zTGmNPESMJ9\nL5A+5K1PbRvIUoYoyajq46q6QFUX1NSM4kyMyQRsfA5mXE1bvIBVWw7xsfPq8brz++YgY4w5bCRp\ntxqYISKNIuLDCfAV/Q8SkTLgg8DzmW3iCdi3DnoPwOwbeXHjARJJZcm8idlulTHGnDLDhruqxoHP\nAS8Bm4CnVXWjiNwjIvekHfpR4LeqGhydpr4P2/4ICEy7gpV/28/UmiJmjrcnUo0xY8eIVmJS1ZXA\nyn7bHuv3+ofADzPVsJOy7Y8w4RxatZjXt7Vx3+XT7d52Y8yYkn9F6Ei3MwPktCt4ccMBkgrXzZ2Q\n7VYZY8wplX/hvvPPkIwfKclMqynizForyRhjxpb8C/dtfwRvES3l5/DG9jauO3uClWSMMWNOfoZ7\nw8W8uKUjVZKxu2SMMWNPfoV7x05o3+aUZNbvZ/q4Ys6oLc52q4wx5pTLr3DftgqAzomX8OaONq61\nkowxZozKs3D/I5TWs7qniqTCJTOqs90iY4zJivwJ92TCWQh72uWs3d2J1y2cXVeW7VYZY0xW5E+4\nH9wIkS6Yehlv7e5g9sQyCrzubLfKGGOyIn/CvfU9AGLVM1nf3Mn8yRVZbpAxxmRP/oR7WxMgbO6r\nJhJLct6U8my3yBhjsia/wr18Emv2hgGYP8VG7saYsSt/wr11K1RN563dnUwoK2BCWWG2W2SMMVmT\nH+GuCm3bnHDf1cF5Nmo3xoxx+RHuvYcg2kN3UQN7O8OcZxdTjTFjXH6Ee1sTAJtjzhqpVm83xox1\neRLuWwF4s6sSv8fF7AmlWW6QMcZk14hWYjrttTWB28+qAz7m1hfi8+TH7yxjjDlR+ZGCbdtIVk5l\nw75eq7cbYwwjDHcRWSwiW0SkSUQeGOSYy0RknYhsFJE/ZbaZw2jdSm9RA9FEkrPrbT4ZY4wZtiwj\nIm7g28DVQDOwWkRWqOq7aceUA48Ci1V1t4iMG60GHycRh44ddI+/EoCaYv8pO7UxxpyuRjJyXwg0\nqep2VY0Cy4El/Y65A/iFqu4GUNVDmW3mEDp3QTJOW8FkAMoDvlN2amOMOV2NJNzrgD1pr5tT29Kd\nAVSIyMsislZEPpmpBg6rbRsAB7z1AJQVek/ZqY0x5nSVqbtlPMB84EqgEHhdRN5Q1ffSDxKRZcAy\ngMmTJ2fmzKnbIPe6JgKHKA9YuBtjzEhG7nuBSWmv61Pb0jUDL6lqUFVbgVeAc/q/kao+rqoLVHVB\nTU3Nibb5WG1NUFjBwXgxfo/L5nA3xhhGFu6rgRki0igiPmApsKLfMc8DF4uIR0QCwPnApsw2dRBt\nTVA1nc5QzEbtxhiTMmxZRlXjIvI54CXADTyhqhtF5J7U/sdUdZOIvAisB5LA91R1w2g2/Ii2bdB4\nKZ29Uau3G2NMyohq7qq6EljZb9tj/V5/Hfh65po2AtEgdO+Fqml0tcQoL7Q7ZYwxBnL9CdXUnTJU\nzaAzFKPbXUc4AAANpElEQVTMyjLGGAPkerj3pm6nL51IVzhGuZVljDEGyPVwj/Y6f/qK7IKqMcak\nye1wj4UA6HMVEI4l7IKqMcak5Ha4R4MAdCedC6llNvWAMcYAeRLuXTEn1K3mbowxjtwO91gIEDpi\nzlOpVnM3xhhHbod7NAjeAJ3hBGCThhljzGG5H+6+AF3hGIA9xGSMMSl5EO5FdIaiAPYQkzHGpOR2\nuMdC4C2iKxzDJVDiz4/1vo0x5mTldrgfGbnHKC304nJJtltkjDGnhTwI94BNPWCMMf3kdrinyjKd\n4Zg9wGSMMWlyO9xTZZmuUNRG7sYYkyYPwj3gjNwt3I0x5ojcDvdYCHzFNiOkMcb0k7vhrgrRIOop\npDtiF1SNMSZd7oZ7LAwoEVchqjYjpDHGpBtRuIvIYhHZIiJNIvLAAPsvE5EuEVmX+vhK5pvaT2ou\n9xB+wOaVMcaYdMM+0ikibuDbwNVAM7BaRFao6rv9Dn1VVa8fhTYOLLUKUzDphLuVZYwx5qiRjNwX\nAk2qul1Vo8ByYMnoNmsEos7IvVdT4W4XVI0x5oiRhHsdsCftdXNqW38Xish6EXlBROYM9EYiskxE\n1ojImpaWlhNobprDqzAlUgt1WLgbY8wRmbqg+hYwWVXnAv8H+OVAB6nq46q6QFUX1NTUnNwZY6lV\nmOJOqJdaWcYYY44YSbjvBSalva5PbTtCVbtVtTf1+UrAKyLVGWvlQFJlmY54av1UC3djjDliJOG+\nGpghIo0i4gOWAivSDxCR8SIiqc8Xpt63LdONPUaqLNMe8xDwufF73KN6OmOMySXD3i2jqnER+Rzw\nEuAGnlDVjSJyT2r/Y8DNwL0iEgfCwFJV1VFs95GyTGvUS3mhTfVrjDHpRrS6RarUsrLftsfSPv8v\n4L8y27RhpEbuLREPpYW5+yyWMcaMhtxNxVTNvaXPZXfKGGNMP7kb7rEguH20R9QWxjbGmH5yN9yj\nQfAGbEZIY4wZQA6Hewj1Fdtc7sYYM4AcDvde1FtINJ6kzEbuxhhzjNwN91iIuLsQwGruxhjTT+6G\nezRE3B0AbF4ZY4zpL4fDvZc+VwFgUw8YY0x/uRvusRBRl1OWKfKP6FksY4wZM3I33KNB+iQV7j6b\nV8YYY9LleLg7C3UUWrgbY8wxcjfcYyHCqZF7wGdlGWOMSZeb4Z6IQSJ6ZHHsgI3cjTHmGLkZ7qkZ\nIUP4cQn4PbnZDWOMGS25mYqpcA8m/RT5PKTWCTHGGJOSm+Eec6b77VW/XUw1xpgB5Ga4p0bu3Um/\n1duNMWYAOR3uPQmv3SljjDEDGFG4i8hiEdkiIk0i8sAQx31AROIicnPmmjiAVFmmM24jd2OMGciw\n4S4ibuDbwDXAbOB2EZk9yHH/Cfw20408Tmrk3pXwErCpB4wx5jgjGbkvBJpUdbuqRoHlwJIBjvsf\nwLPAoQy2b2CHwz3uIeC1kbsxxvQ3knCvA/akvW5ObTtCROqAjwLfyVzThpAqy7RFfVaWMcaYAWTq\nguojwBdVNTnUQSKyTETWiMialpaWEz9btBeAjpiXgN/C3Rhj+htJwXovMCntdX1qW7oFwPLUw0TV\nwLUiElfVX6YfpKqPA48DLFiwQE+00URDgNAec9ndMsYYM4CRJONqYIaINOKE+lLgjvQDVLXx8Oci\n8kPg1/2DPaNiIdRXRKRbKbSauzHGHGfYcFfVuIh8DngJcANPqOpGEbkntf+xUW7j8aK9qNdZYq/I\nyjLGGHOcEdU0VHUlsLLftgFDXVU/dfLNGkY0RNLjhHuhlWWMMeY4OfuE6uFwt1WYjDHmeLkZ7rEg\n8VS4262QxhhzvNwM92iImLsAsLKMMcYMJEfDPUjMZYtjG2PMYHIz3GNB+lLhbvO5G2PM8XIz3KNB\n+sQpyxRZWcYYY46To+EeIpIKd7ugaowxx8u9cE8mIRYizOELqhbuxhjTX+6FezwMKCH8ADa3jDHG\nDCD3wj3qTPcbUj9+jwu3S7LcIGOMOf3kYLg70/32qp8iW4XJGGMGlHvhnlqooyfptxkhjTFmELkX\n7qmyTE/CVmEyxpjB5GC4O2WZroTPFsc2xphB5F64p8oyXQmvLY5tjDGDyL1w9/hh3Gza4oVWljHG\nmEHkXrhPvwo++zo7EjVWljHGmEHkXrinhKJxK8sYY8wgcjfc+xIEbP1UY4wZ0IjCXUQWi8gWEWkS\nkQcG2L9ERNaLyDoRWSMiF2e+qUepKqFYwmruxhgziGGL1iLiBr4NXA00A6tFZIWqvpt22B+AFaqq\nIjIXeBqYORoNBogmkiSSavPKGGPMIEYycl8INKnqdlWNAsuBJekHqGqvqmrqZRGgjKJQXwKw6X6N\nMWYwIwn3OmBP2uvm1LZjiMhHRWQz8Bvg05lp3sBCMQt3Y4wZSsYuqKrqc6o6E7gR+LeBjhGRZama\n/JqWlpYTPlc4Ggdsul9jjBnMSMJ9LzAp7XV9atuAVPUVYKqIVA+w73FVXaCqC2pqat53Yw8LWlnG\nGGOGNJJwXw3MEJFGEfEBS4EV6QeIyHQRkdTn5wF+oC3TjT0sFHXC3VZhMsaYgQ1b11DVuIh8DngJ\ncANPqOpGEbkntf8x4GPAJ0UkBoSB29IusGZcOOaUZWxxbGOMGdiI0lFVVwIr+217LO3z/wT+M7NN\nG5yVZYwxZmg5+YRq2MoyxhgzpJwM92DUyjLGGDOUnAx3u6BqjDFDy8lwD0cTuF2C35OTzTfGmFGX\nk+kYTE33m7r70hhjTD85Ge7haMJKMsYYM4ScDPdQNEGRrcJkjDGDytFwj1NoqzAZY8ygcjTcExTZ\nKkzGGDOonAz3YDRBod3jbowxg8rJcA/b4tjGGDOknAz3UNQWxzbGmKHkbrjbrZDGGDOoHA33uK3C\nZIwxQ8i5cE8klUgsaSN3Y4wZQs6Fe9gWxzbGmGHlXLiHbHFsY4wZVs6F++GFOmzkbowxg8u5cLcl\n9owxZngjCncRWSwiW0SkSUQeGGD/x0VkvYj8TUReE5FzMt9Ux+HFsa0sY4wxgxs23EXEDXwbuAaY\nDdwuIrP7HbYD+KCqng38G/B4pht6mI3cjTFmeCMZuS8EmlR1u6pGgeXAkvQDVPU1Ve1IvXwDqM9s\nM48KHam528jdGGMGM5JwrwP2pL1uTm0bzGeAF06mUUOpKfFxzVnjqSzyjdYpjDEm52V0+Csil+OE\n+8WD7F8GLAOYPHnyCZ1j/pRK5k+pPNEmGmPMmDCSkfteYFLa6/rUtmOIyFzge8ASVW0b6I1U9XFV\nXaCqC2pqak6kvcYYY0ZgJOG+GpghIo0i4gOWAivSDxCRycAvgE+o6nuZb6Yxxpj3Y9iyjKrGReRz\nwEuAG3hCVTeKyD2p/Y8BXwGqgEdFBCCuqgtGr9nGGGOGIqqalRMvWLBA16xZk5VzG2NMrhKRtSMZ\nPOfcE6rGGGOGZ+FujDF5yMLdGGPykIW7McbkoaxdUBWRFmDXCX55NdCawebkirHY77HYZxib/R6L\nfYb33+8pqjrsg0JZC/eTISJrxuKtlmOx32OxzzA2+z0W+wyj128ryxhjTB6ycDfGmDyUq+E+avPF\nn+bGYr/HYp9hbPZ7LPYZRqnfOVlzN8YYM7RcHbkbY4wZQs6F+3DrueYDEZkkIqtE5F0R2Sgin09t\nrxSR34nI1tSfFdlua6aJiFtE3haRX6dej4U+l4vIMyKyWUQ2icgFY6Tf/5j6+71BRH4qIgX51m8R\neUJEDonIhrRtg/ZRRL6UyrYtIvLhkzl3ToX7CNdzzQdx4J9UdTawCLgv1c8HgD+o6gzgD6nX+ebz\nwKa012Ohz98EXlTVmcA5OP3P636LSB1wP7BAVc/CmXF2KfnX7x8Ci/ttG7CPqX/jS4E5qa95NJV5\nJySnwp0RrOeaD1R1v6q+lfq8B+cfex1OX3+UOuxHwI3ZaeHoEJF64DqcRV8Oy/c+lwGXAt8HUNWo\nqnaS5/1O8QCFIuIBAsA+8qzfqvoK0N5v82B9XAIsV9U+Vd0BNOFk3gnJtXB/v+u55jwRaQDOBd4E\nalV1f2rXAaA2S80aLY8A/zeQTNuW731uBFqAH6TKUd8TkSLyvN+quhd4GNgN7Ae6VPW35Hm/Uwbr\nY0bzLdfCfUwRkWLgWeALqtqdvk+d25zy5lYnEbkeOKSqawc7Jt/6nOIBzgO+o6rnAkH6lSLysd+p\nOvMSnF9uE4EiEbkz/Zh87Hd/o9nHXAv3Ea3nmg9ExIsT7E+q6i9Smw+KyITU/gnAoWy1bxRcBHxE\nRHbilNuuEJGfkN99Bmd01qyqb6ZeP4MT9vne76uAHaraoqoxnGU6LyT/+w2D9zGj+ZZr4T7seq75\nQJy1Cr8PbFLVb6TtWgHclfr8LuD5U9220aKqX1LVelVtwPm5/lFV7ySP+wygqgeAPSJyZmrTlcC7\n5Hm/ccoxi0QkkPr7fiXOtaV87zcM3scVwFIR8YtIIzAD+OsJn0VVc+oDuBZ4D9gG/HO22zNKfbwY\n579q64F1qY9rcdap/QOwFfg9UJntto5S/y8Dfp36PO/7DMwD1qR+3r8EKsZIvx8ENgMbgB8D/nzr\nN/BTnGsKMZz/pX1mqD4C/5zKti3ANSdzbntC1Rhj8lCulWWMMcaMgIW7McbkIQt3Y4zJQxbuxhiT\nhyzcjTEmD1m4G2NMHrJwN8aYPGThbowxeej/B4GwDk6CgNcXAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1f06eac0d30>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['training', 'validation'], loc = 'upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training and validation accuracy improve consistently, but reach plateau after around 60 epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 9504/10000 [===========================>..] - ETA: 0s"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy:  0.9154\n"
     ]
    }
   ],
   "source": [
    "print('Test accuracy: ', results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Dropout (Regularization)\n",
    "- Dropout is one of powerful ways to prevent overfitting\n",
    "- The idea is simple. It is disconnecting some (randomly selected) neurons in each layer\n",
    "- The probability of each neuron to be disconnected, namely 'Dropout rate', has to be designated\n",
    "- Doc: https://keras.io/layers/core/#dropout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://image.slidesharecdn.com/lecture29-convolutionalneuralnetworks-visionspring2015-150504114140-conversion-gate02/95/lecture-29-convolutional-neural-networks-computer-vision-spring2015-62-638.jpg?cb=1430740006\" style=\"width: 500px\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.layers import Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def mlp_model():\n",
    "    model = Sequential()\n",
    "    \n",
    "    model.add(Dense(50, input_shape = (784, )))\n",
    "    model.add(Activation('sigmoid'))    \n",
    "    model.add(Dropout(0.2))                        # Dropout layer after Activation\n",
    "    model.add(Dense(50))\n",
    "    model.add(Activation('sigmoid'))\n",
    "    model.add(Dropout(0.2))                        # Dropout layer after Activation\n",
    "    model.add(Dense(50))\n",
    "    model.add(Activation('sigmoid'))    \n",
    "    model.add(Dropout(0.2))                        # Dropout layer after Activation\n",
    "    model.add(Dense(50))\n",
    "    model.add(Activation('sigmoid'))    \n",
    "    model.add(Dropout(0.2))                         # Dropout layer after Activation\n",
    "    model.add(Dense(10))\n",
    "    model.add(Activation('softmax'))\n",
    "    \n",
    "    sgd = optimizers.SGD(lr = 0.001)\n",
    "    model.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = ['accuracy'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = mlp_model()\n",
    "history = model.fit(X_train, y_train, validation_split = 0.3, epochs = 100, verbose = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXmYJGWd579v3ndl3WfT1Q19dwNNH7QiCALKoYCoiI4H\nMyorOyPjsM7ajjOis7rjuozjOoMXiuiMI4M4KoPcCgIK0tVA33dT3V1dlXXmfUVkxrt/RLyREZmR\nmZFHHd31fp6HpyojIzIiC3i/7+8mlFJwOBwOh2OZ7wfgcDgczsKACwKHw+FwAHBB4HA4HI4CFwQO\nh8PhAOCCwOFwOBwFLggcDofDAcAFgcPhcDgKXBA4HA6HA4ALAofD4XAUbPP9ALXQ0dFBBwcH5/sx\nOBwO54xi586dU5TSzmrnnVGCMDg4iKGhofl+DA6HwzmjIIScMHMedxlxOBwOBwAXBA6Hw+EocEHg\ncDgcDoAzLIZghCiKGBkZQSaTme9HOStwuVwYGBiA3W6f70fhcDhzzBkvCCMjI/D7/RgcHAQhZL4f\n54yGUorp6WmMjIxg2bJl8/04HA5njjnjXUaZTAbt7e1cDJoAIQTt7e3c2uJwFilnvCAA4GLQRPjf\nksNZvJwVgsDhcDjzxY7hGewfjc33YzQFLggNEolE8K1vfavm66677jpEIpGK53zhC1/AM888U++j\ncTicOeDzv9iDL/96/3w/RlPggtAg5QQhl8tVvO6xxx5DMBiseM7f//3f46qrrmro+TgczuwynRBw\neDw+34/RFLggNMj27dtx7NgxXHjhhdiyZQsuvfRS3HDDDVi7di0A4KabbsKmTZuwbt06fO9731Ov\nGxwcxNTUFIaHh7FmzRp84hOfwLp16/D2t78d6XQaAHDbbbfh4YcfVs+/++67cdFFF2HDhg04ePAg\nAGBychJXX3011q1bh49//ONYunQppqam5vivwOEsTiSJIpIWMZUQMJ3IzvfjNMwZn3aq5Uv/ta/p\nvry1fQHc/a51Zd//6le/ir179+L111/Hc889h+uvvx579+5V0zbvv/9+tLW1IZ1OY8uWLXjPe96D\n9vZ23WccOXIEP/3pT3Hffffhlltuwc9//nN86EMfKrlXR0cHXn31VXzrW9/CPffcg+9///v40pe+\nhLe97W343Oc+hyeeeAI/+MEPmvr9ORxOeeKZHPISBQAcHk/gTT7nPD9RY3ALocls3bpVl8P/zW9+\nExdccAG2bduGU6dO4ciRIyXXLFu2DBdeeCEAYNOmTRgeHjb87JtvvrnknBdffBG33norAOCaa65B\na2trE78Nh8OpRDglqL+fDW6js8pCqLSTnyu8Xq/6+3PPPYdnnnkGL730EjweDy6//HLDHH+ns7Cr\nsFqtqsuo3HlWq7VqjILD4cw+WkE4dBYIArcQGsTv9yMeN/4PIRqNorW1FR6PBwcPHsTLL7/c9Ptf\ncskleOihhwAATz31FMLhcNPvweFwjGGC4LZbcaQOQdh1KoK/+PdXVbfTfMMFoUHa29txySWXYP36\n9fjrv/5r3XvXXHMNcrkc1qxZg+3bt2Pbtm1Nv//dd9+Np556CuvXr8fPfvYz9PT0wO/3N/0+HA6n\nlHBSBABsHmzFoVAclNa2sL94dAqP7h5bMAHps8plNF/8+7//u+Fxp9OJxx9/3PA9FgPo6OjA3r17\n1eOf+cxn1N8feOCBkvMBYPPmzXjuuecAAC0tLXjyySdhs9nw0ksvYceOHToXFIfDmT2YhXDxsja8\ncGQK47Eselpcpq+PZ2TXbzQtoitg/rrZggvCGc7Jkydxyy23QJIkOBwO3HffffP9SBzOoiGcEmAh\nwEVL5WSOw+PxmgQhmS0IwkKAC8IZzooVK/Daa6/N92NwOIuScEpEq8eB1T0BALIgXLay6uhilcQC\nEwQeQ+BwOJw6iaQEBD12tHkd6PA5cShUW2BZ6zJaCHBB4HA4nDqZSQpo9TgAAKt6fDg8kajp+kRW\nFgIuCBwOh3OGE0mJaPXKgrCiy48j43FINaSQJrN5AEAsvTDqirggcDgcTp2EUwJaPfK42VU9fqSE\nPE5HSgtLH98zhi8/WtoRlccQFjk+nw8AMDo6ive+972G51x++eUYGhqq+Dnf+MY3kEql1Ndm2mlz\nOJzmQSlVg8oAsLJbrv8xamHx45dO4MEdp0qO8xgCBwDQ19endjKth2JBMNNOm8PhNI+UkIeQkwou\no255s1fcwkLMS3j9VASJbA65vKR7b6GlnXJBaJDt27fj3nvvVV9/8YtfxJe//GVceeWVaqvqX/3q\nVyXXDQ8PY/369QCAdDqNW2+9FWvWrMG73/1uXS+jO+64A5s3b8a6detw9913A5Ab5o2OjuKKK67A\nFVdcAaDQThsAvv71r2P9+vVYv349vvGNb6j3K9dmm8Ph1A4rSmMuo4DLjr4WF46M6wPLB8ZiSIty\nrIC5iAAgl5fU47EFIghnVx3C49uB0J7mfmbPBuDar5Z9+/3vfz8+/elP48///M8BAA899BCefPJJ\n3HnnnQgEApiamsK2bdtwww03lJ1X/O1vfxsejwcHDhzA7t27cdFFF6nvfeUrX0FbWxvy+TyuvPJK\n7N69G3feeSe+/vWv49lnn0VHR4fus3bu3Ikf/vCH+OMf/whKKS6++GK89a1vRWtrq+k22xzOmczQ\n8AweGjqF//Oe82d1RngkJS/iQcVlBMjt8odOzIBSqt57aLjQXyyWzqnns4AywC2Es4aNGzdiYmIC\no6Oj2LVrF1pbW9HT04O/+Zu/wfnnn4+rrroKp0+fxvj4eNnPeP7559WF+fzzz8f555+vvvfQQw/h\noosuwsaNG7Fv3z7s3195VN+LL76Id7/73fB6vfD5fLj55pvxwgsvADDfZpvDOZN5ev84HhoamfVF\ndiYpWwht3oIgXLG6C6dm0jissRJ2ntAIQqbwTAlBthYsRH98Pjm7LIQKO/nZ5H3vex8efvhhhEIh\nvP/978dPfvITTE5OYufOnbDb7RgcHDRse12NN954A/fccw927NiB1tZW3HbbbXV9DsNsm20O50xm\nUmkUN5XI6nbvzabYZQQAV63pxud/sRfPHBjHqh4/KKUYOjGD7oAT47GszjWUUALKPQEXImeShUAI\nuYYQcogQcpQQst3g/dWEkJcIIVlCyGeK3rufEDJBCNlbdPyLhJDThJDXlX+ua+yrzB/vf//78eCD\nD+Lhhx/G+973PkSjUXR1dcFut+PZZ5/FiRMnKl5/2WWXqQ3y9u7di927dwMAYrEYvF4vWlpaMD4+\nrmuUV67t9qWXXopf/vKXSKVSSCaT+MUvfoFLL720id+Ww1nYTMZlQZiIz24HUSOXUXfAhQsGWvDU\nftkjMBJOYzyWxRWrugAUWQhKUVpf0I2UkIdYFHCeD6oKAiHECuBeANcCWAvgA4SQtUWnzQC4E8A9\nBh/xAIBrynz8P1FKL1T+ecz0Uy8w1q1bh3g8jv7+fvT29uJP/uRPMDQ0hA0bNuDHP/4xVq9eXfH6\nO+64A4lEAmvWrMEXvvAFbNq0CQBwwQUXYOPGjVi9ejU++MEP4pJLLlGvuf3223HNNdeoQWXGRRdd\nhNtuuw1bt27FxRdfjI9//OPYuHFj8780h7NAmUoIup+zBXMZBd123fGr13Zj16kIJmIZ1V10xWpF\nEDQFaAklhtAXdANYGHEEMy6jrQCOUkqPAwAh5EEANwJQndmU0gkAE4SQ64svppQ+TwgZbMrTLmD2\n7CkEszs6OvDSSy8ZnpdIyL7FwcFBte212+3Ggw8+aHi+tgW2lk996lP41Kc+pb7WxgPuuusu3HXX\nXbrztfcD9G22OZyziSnmMpp1C0FAwGWDzarfV1+1thv3PHUYzxyYwP6xKHxOG7YOtgEoshAUl1F/\na0EQOuZ5JrMZl1E/AG1FxYhyrBl8ihCyW3ErGQ4DJoTcTggZIoQMTU5ONum2HA5nNommRdzxbzvV\nxXmukCSq7tzrubeYl3DnT1/D0YnqTerCmrYVWlZ1+7GkzY1nDoxjaDiMjecE0eK2gxB9eilzGfUr\nFsJCSD2dzyyjbwNYDuBCAGMA/tHoJErp9yilmymlmzs7zbeV5XA488e+0Sge3xvCayfntno+nBLU\ncZT1CMJoJI1Hdo3i2YPVN59y24pSQSCE4Oo1PXjx6BQOjcexaWkrLBYCn9OGWKbgMmJVyv0LyGVk\nRhBOA1iieT2gHGsISuk4pTRPKZUA3AfZNVXvZzX6OBwF/rfkNIO0IPvHk9m5bdqmjRvUE0NgtQET\n8erZfNo+RsVctbYLQk4CpcDmpbK7KOCy61xG7F69QXmgzpkiCDsArCCELCOEOADcCuCRRm9MCOnV\nvHw3gL3lzq2Ey+XC9PQ0X8iaAKUU09PTcLnmf5Qf58zGqDJ3LmBWgdturctCSCm1AeOx6teGk6Kh\nhQAAWwbb0OK2w0KAC8+RW8oE3PaioLIIt92q1jEsBJdR1aAypTRHCPkLAE8CsAK4n1K6jxDySeX9\n7xBCegAMAQgAkAghnwawllIaI4T8FMDlADoIISMA7qaU/gDA1wghFwKgAIYB/Ld6vsDAwABGRkbA\n4wvNweVyYWBgYL4fg3OGk5o3C0FeyFf1+DERq71mJymYtxAiKcEwhgAAdqsFN1/Uj+OTSfic8jIb\ncNmK0k5z8DptaFGylBaChWCqME1JCX2s6Nh3NL+HILuSjK79QJnjHzb/mOWx2+1YtmxZMz6Kw+E0\niYw4P4LAahDW9PqxfzSmayFhhpTyvBNVLIRsLo+kkC/rMgKAu9+1Tvc64Lbj1EyhIWUim4ffZYPT\nZoXLblkQgsBbV3A4nKbDLISEpl/PXDCZyMJhtWBZhxdCXqp58AyzEMarWBdGRWnVCLjsaiAZABIZ\nUbUeWorcSfMFFwQOh9N05s1lFBfQ7nOg0y/n80/WGEdgMYSkkK8Y/2BtK9rKuIyM8LtsRWmnOZ0g\ncAuBw+GclTCXEWvgNldMJbLo8DnR6XOpr2tB24G0UgxCrVKu4DIqJuC2I57NqWmxiWweXjW+wAWB\nw+Gcpag77XkIKnf6nejwO9TXtZDSCFilXkjMZVQuy8iIgEte/FmFciIrwu/iFgKHwznLSQtyo7b5\nEIQOn0NtAVFr+wqthVApjlCPyyigZBOxTKNEhruMOBzOIiAtykKgDaLONpJEMZ0Q0OFzotXjgIXU\nXpyWFnNw260AKmcaFYLKNbiMXPr00qTWZeS2L4iZCFwQOBxO01GDynMYQ4imReQkig6fE1YLQZvX\nWVcMoTvghMtuqViLMJMU4HFY4bRZTX92wC0v/rGMiGwuDyEv6VxG8UwhvjBfcEHgcDhNp9C6Yu7S\nTtni36FkGHX4HHXFELxOG7oDrorVyuX6GFWCWQixdE6NI/g0FgIAxOfZSuCCwOFwms58tK5gKaYd\nPnmh7vQ7MVmjyyiZzcPrsKHL76xoIURSIlq95t1FANSK5FhGVIXSq4khAPNfrcwFgcPhNB3mMhJy\n0pxNAmPxgk4fsxCcNQeVU0IObocVXQFXxRjCTLJ+CyGeySGutL72cUHgcDhnO8xlBMxdphFrW8Ey\njGQLIVtT48ukkIfXaUW331U27ZRSiqlEtmZB8CnxglhaVF1G2hgCwAWBw+GchaTFPBzKJLG5chtN\nJbKwWYi6uHb4HBByEuI13D+VzcHjsKEr4EQimzN89mcOTGAknMaWQcOZXmWxWgj8TrnBHfvcYgth\nvttXcEHgcDhNJy3k1fYRcxVYnopn0e5zwGKRm9nVU4uQFPLwOqzoDsjXFlcrZ3N5fOXX+3Felw+3\nbj2n5mdkLbCZIBTSTuWfRhbCG1NJ3HTv7zE0PFPz/WqFCwKHw2kqkkSRFguCMJcWgnYmsSoINQSW\nU0IOHqcN3X659UWx2+iB3w9jeDqFv3vnWtittS+ffpfeQjDjMjo+mcDrpyKwWsx3ba0XLggcDqep\nZHKyRcAW5LmKIUwpRWmMgiCYsxDkADiF12FFl2IhaKuVJ+NZ/PNvj+Jtq7vw1pX1jfOVLQSxJO3U\nbbfCbiWGgjA8LbfMXtrureuetcAFgcNZhOwbjeLEdHJWPpsFlAsuo7mzENg9AdTcz4g9txxDkC2E\nSY2FcM+Th5AR8/jb69fU/YzykJwcktkcCAE8DrmwjRBStn3Fiekk/C5bxdkLzYILAoezCLnj317F\n/3ni4Kx8Nks57VTqAebCZURpoW0Fo83jACHmYwisqtrrtMLvtMFlt6gWwkxSwMOvjuBD25Zieaev\n7ucMuGQLIZ7Nweew6Yb3lGtfMTydwmC7t6ZBP/XCBYHDWWREUgJOzqQQTs5OiiMrSpvLGEIsnYOQ\nl9SiNACwWS1o8zhMF6exTqceZaHWVis/tS+EvETxvs2NjZdli34ik1PTUBktijupmJPTSSxt9zR0\nX7NwQeBwFhn7R2MAoBZHNZv5cBmxKmWtywhQitNMuowK1cOyG0euRZAthF/vGcPSdg/W9gYaes6A\ny4ZENoe4ptNp4b1Sl5GYlzASTnNB4HA4s8M+RRASs9SJlLmMAm477FZieozmayfDFYfSVELtY+TT\nC0Kn36mLA1QiqbEQAKAz4MRELItwUsAfjk3j2vW9DbttAm47KAXGYhk15ZRhFEMYjaSRk+icBJQB\nLggczqJj32gUwOy5cti0NLfdCq/TZtpCuO2HO/Dt3x2r657FVcqMWhrcpZiFoAgCq1Z++sA48hLF\ndRt66no2Lax9xWgkraacMowE4YSSYTTIBYHD4cwGzEKIzbKF4HHY4HWYE4S0kEc0LaqjKWtlqqix\nHYO5jIrbV0gSxW8OjOuOMwvBrWT+dCvVyg8PjWCg1Y0N/S11PZsWVoA2lciWuIxYDEHStMBmmWCD\n3GXE4XCaTVrI49hkAi67BUJOQjbX/CriQnDWCp/TZsoSYQu6UVDVDFOJLKwWUtJfqMPvREaUkBT0\n3/Pl49P42I+GMHQirHlufQyB1SK8MjyD6zY07i4CChYCpTB0GUlUP0NieDoFt91aEhuZLUgtjZ/m\nm82bN9OhoaHaL3x8OxDa0/wH4nDOMOJZEftGY3L6Y0bEpnNa66q4rUQolsHwdBIXndOKw+NxWAip\nGoxlz+Vz2rC+r/ad+PGpBGaSAjYvbdMdn4hncHwqiQuXBOHSDLOZTGRxbDKBFV0+tHvlxXYsmsaJ\nmRQ2L22FzWJBNC3iQEi2ptb1BeB3Nl4HkBRy2HNadtn1BFw6VxB71gsGgurUtkPjMWRFCecPBIGe\nDcC1X63rvoSQnZTSzdXO4xYCh7OIYH5yNvA9PwsbQkn5TKuFwGohpu4h5uVz6p0YlpMobAbCxsSu\nuAV3TnnN7gsU/hasF5LDpvy0WkrcO/WibT9R3IqC3UM7JCcjSnDazU9laxhK6Rnzz6ZNmyiHw6mf\n7T/fTc//4pP0ib1jdOlnH6V7RiJNv8c/PX2ILv3sozSXl+gd/zZEr/zH56pe8+9/PEGXfvZRuvUr\nT9d1zz+572V6070vlhx/9cQMXfrZR+kz+0O64//3iYN06Wcfpd94+rB67B8eO0BXfP4x9XUsLdBl\n2x+lX3pkX13PZEQ4maVLP/soXfrZR+m3nzuqe0+SJLrly0/TP//JTkoppfm8RFd8/jH6lV/vb/i+\nAIaoiTWWWwgcziJi/2hUdn+42G60scDyqZkUPvefe3Q78LSQh9NmgdVCTAeVp9UYQum5P/njCfzH\njpMVr4+mRQTdpS4d5g4qDlbPpATleCEDKSXk4HUUduN+lx0//rOL8emrV1R9frNoLY1iq4MQgres\n6MAfjk1DkihCsQyEnDRnNQgAdxlxOIsGMS/hQCiu84c3mnr624MT+OkrJ3V9kVJCXu3R4zUdVJYX\n6LSYL3Hv/OtLJ/Cl/9qPSKp8BlIkLagdQ7W0KVlHJYKg3G8mVXDPJLN5tQaB8ZYVHWoguBnYNO6n\n4rRTALh0RQdmkgL2j8UwrGYYzU3KKcAFgcNZNBybTEDISVjX16IuRokGq5XZzj6sWVjTYl4NivqU\nOgRaJY6grRUotloiKREpIY9/e/lE2eujKdFQELwOKxw2S1kLIaw5nhJyaobRbMLiN0ZxiUvO6wAA\nPH9kEifVLqfcQuBwOE1m3+lCxoyvSS4j1idIu7Cmhbyay+912iDRQn+jcmgFoTj1NKws3g/8YVgt\netOSlyji2RxaDEZaEkLQ5nFgukgQ2PNqjyeFUgthNvArFkdx2ikAdPldWN3jx4tHpjA8nYLdStDb\n4p71Z2JwQeBwFgn7RmNw2S1Y3unTZLQ0JgjMQohoFnE2qB4AfMqOu5rbaCohwGmTl6OYLssmj2xO\nwmUrOzGVEPDzV0dKro1nRFAKQwsBANq8Dp1gAQUXks5CyM6RheAubyEAsttoaDiMg6EYlrR55mQw\nDoMLAoezSNg3GsXqngCsFgKX3QqH1dK4ICgLqta/nxbz8NjlxY7tgquN0ZxOZLGsQ/aVawPLEcUV\ndc26Hlww0IL7nj9ekprK2j2UE4R2n95CkCSqWh0zKUF1Z82VhcBiEkYxBAB4y4pOCHkJLxyZmtP4\nAcAFgcNZNByZSGBNr1997XPZZieGUOQyAip3PBXzEsIpsSAIGguBLdytHjtuv+xcDE+n8NS+kO56\nJghGWUbytQ5dDCGWESFRuTBMyBWqmIuzjGaLgLu8ywgAtg62wWGzIC/ROY0fAFwQOJxFQUbMYyYp\noD9Y8Ef7nLaGO55OJ5iFUD6oDFR2GbHFenknsxAKn8U+t8VjxzXre3BOmwf/9kd9cFl7jhHFLiNm\nLZzXJQ+6Ye8ls3l4mlSAVolKQWVA7qW0ZbAVALC0jQsCh8NpMmzyV48mQOl32RpyGWXEPOLKQq91\nGRWnnQKVLQQWUF7WIS/QWgsholoIDlgtBBedE1Q7gDKquoy8DsSzObVvExOAcxUBYoKUEnLwzEFV\n8IpuP/qDbjVmYsRbzpNnNi/tmFuX0ezLIYfDmXfGorIg9La41GM+p01d0OtB65cPp/RZRi6HeQuB\n1SAsbffAQopiCMwdpOz+O3xO1SphVHUZeeXso3BSRE+LVRUAZiHMJAVIEpVjH3NgIfzJxefgg1vP\nqdgs790b+7FvNIpNS1tn/Xm0cAuBw1kEhKLMQigIgt9lb8hCYPEDu5WUuIw8RS6jSkFlNvO40+cs\nmSvMhCbolhf1dp8TaTGvsziYIAQqWAhAwRJgP8/VCEIml5c7kM5BDIEQovZLKkdPiwv/8sGLmloU\nZwZTgkAIuYYQcogQcpQQst3g/dWEkJcIIVlCyGeK3rufEDJBCNlbdLyNEPI0IeSI8nNupZDDWUQw\nC6EnoBWExoLKbKe+rMOrCgKlyk5bdRnJPyu5jKaV9hHtPoc6hJ4RTYlw2ixqkJrNO9DWLUTT8jmu\nMu6etmJBUETmvE4lhpASVMGaCwthIVNVEAghVgD3ArgWwFoAHyCErC06bQbAnQDuMfiIBwBcY3B8\nO4DfUEpXAPiN8prD4dSBVKVLaCiaRsBl02W2NBpUZnOMz+vyqTv5bE4CpVBdRmz6WDWXkdMmt3QI\nuG26wT2RlKi6iwB5vgG7hhEtOqcYJghMeMJJAS67BZ1+J2wWgumkoM5wmAsLYSFjxkLYCuAopfQ4\npVQA8CCAG7UnUEonKKU7AJRsNyilz0MWjGJuBPAj5fcfAbiplgfncDgye0ai2Pi/nsYLRybLnjMW\nzZRUvLKgcrW2EuVgFsJ5nT5kcxLSQr4wLU3ZrVssBB6HtbIgxLPo8DlBCEHAZde1fw6nBN3Qmw4v\nE4SChVCujxGjTY0hFKqT273y/VqVDCTVQpiDOoSFjBlB6AdwSvN6RDnWKN2U0jHl9xCA7iZ8Joez\nqIimRNzxk52IpkUcnUiUPS8Uy+jiB4Bch5CTKLI5qcxVlZlOZOFxWNGrpLJG0oJmWlphYa02V3kq\nKaiuINllpA8qaxf7Dr+xy6iSIAQ9DhCir05u9crnt3vlGgXVQpiDSuWFzIIIKiv9ug23KYSQ2wkh\nQ4SQocnJ8jsgDmexIUkUdz30uppSatQ6miFbCHpB8DfYvmI6KaDd51Cze8JJUe015NK4XqqN0WQW\nAgDFZaRPO9W6g1g7a22mUTSdQ4u7tI8Rw2ohCLrtalbUTEpUrQ5WtJYUuIUAmBOE0wCWaF4PKMca\nZZwQ0gsAys8Jo5Mopd+jlG6mlG7u7Oxswm05nLOD7zx/DL85OIHPX7cGPqd+IdUi5CRMJbIlFgJr\nshYvc101phLyQh5UFtdISihxGQHyrrtaHYIqCEVB5Yhm8QYAh82CgMumtxBSlV1GgFKcpulwyjKP\n2rwOzKQEpLLcQgDMCcIOACsIIcsIIQ4AtwJ4pAn3fgTAR5XfPwrgV034TA5nUXBiOol7njyEd13Q\nh4++eRB+l63sgPrxWAaUAn1FMQQzNQKVmErIvnjmfgkrbaoBqFlBAJQhOcZpp5JEMaNYGoCcOpoU\n8sjlJVBKEUmJJRXIHX5nTS4jQLYsmFUxkxTU2gRWxcwsBC+3ECpDKc0B+AsATwI4AOAhSuk+Qsgn\nCSGfBABCSA8hZATAXQD+lhAyQggJKO/9FMBLAFYpxz+mfPRXAVxNCDkC4CrlNYez6Dk2mcCX/mtf\nyaAYLTuGw5AocOfbzlODseUshFCstAYBQMNT06YTWXT4HGqNQCQtqG2u3SZdRtG0iJxENRZC4ZnS\nYh5CXtJZCIBcnMayjMS83IuoUpYRALR67ZhJCsjm8khkc2hjLiOvA5G0qFpJnkWeZWRKDimljwF4\nrOjYdzS/hyC7koyu/UCZ49MArjT9pBzOIuGJvSH88PfDuGxFJ65Y3WV4zp6RCDwOK5YrufQBt61s\nDMGoShlAQzMRtDt7thhr3TvahdXrtCEpGN+D7fRZOilzY8UyIuyivF8trkDu8DlwMBQHUL1tBaPN\n68TOE2G1XoJNUmvz2EFp4W9UruHcYmFBBJU5HE6B0UgaAPDL18uH6vaclmcjs175FS2EqPx5JRZC\nA2M0Y5nCzt5lt8JttyKcFJBmLiNNDMHnKp9lxHb6HRqXESAHyNUq5WKXkc+pVjebFYR2rwPhlIhJ\n5TqthQCRVSs8AAAgAElEQVQAI+EUCEHF/kKLgcX97TmcBQgThKf2jRsupLm8hP1jMazvb1GPFbd8\n0DIWzcDntKm7b0bBZVR7UJnt7NsVV0/QY0ckLSJVo8tItRCKXEaxjIhoivUxKnUZxTI5CDmpIAhV\nXUYO5CWqzilmQsCylk7NpOF12Cr2F1oMcEHgcBYYY9EMOv1yz56n94+XvH9sMomMKOH8AY0gVOhc\nGoqW1iAABfdIPdXK6s5eWViDHgciKQFpxTWktRC8DhsyooScQUykRBBUC0FUZywUWwgsAD2dzKqi\nYcZCAIBjE0ndaxYQHwmnFn38AOCCwOEsOE5H0rh2fQ/6Wlz4lYHbaPdIBACwodhCSIuGVcdGNQiA\nnMLptFnq6njKMnaYhdDqsSOcEpEW5EVfX5im9DMSSjONphOCWifAvgcgWwiRdKH1tRYmHlNxoYYY\ngvwZRyfl4j1tlhEgZ0gt9vgBwAWBw1lQxDMi4pkc+oNu3HBhP54/MqV2FWXsPR2Fx2FV5wcAcgxB\nosaLbiia0TW101Jvx1NtQzpAXrTDKQEpMQeHzaKbA1wpvXUqkUWb16F2/1RdRulcYfBNSVBZEYRk\ntmrrawZb+I+Mx3Xna8WGWwhcEDicBYWaERR048YL+5CXKH69Z0x3TnFAGSgMbi+uRcjlJUzEjS0E\ngHU8rcNlFM/CQgoLaovHjmhKREbI69xFQOUhOdqiNEB2L1mILIyRlNyErriLqdrxNJ5VRaNc62sG\nE4Q3ppIIeuywWeWlz2W3qg3tFnsNAsAFgcNZULCAcn/QhTW9Aazq9uOXrxXcRkYBZaAwuL04sDyZ\nyMrzg4uK0hhyg7s6gspJAW1ehypKrUpQOamZlsaobCEU+hgBcjM8v8uOWCZXUqXMUC2EhOwy8jlt\nsFsrL2VMELI5Sc0wYjD3kWeRVykDXBA4nAXFaITVDMgL+I0b+/DqyQgOKXn3RycTJQFlQJ+uqaVc\nDQKj3hbY04msmqEDyJZCXqKYiGd1GUZAbRYCwGoq5KCyUWzA67TBbbdiOpE1VaUMyJYAEyomAAwW\nYOYWAhcEDmdBMRZNw2oh6FIKtW7dcg4CLhu+/Ov9oJRiz0gUgD6gDGgshCKXkdGkNC3VGs+VYzpR\naDcBFPz8o5G0gcvIeEgOpVQRBP0CzWoqomnB0EIA5NjFVCKLaFqo6i5itBUFkhmqhcBjCFwQOJyF\nxOlIGt1+p+rjbvM68JdXrcQLR6bw24MThgFlQBNDKHL/MAuhuI8Ro96gcvHOni3cY5F0ycLKAtoj\n4bTu+Hgsi4wooT+ofzbWAjtcYfBNh8+J6aTsMqoWUGaoDe2KRIa95llGXBA4nAXFWCSjzhdgfORN\nS3Fupxdf/vUBvHYqUhJQBjQtH0osBHnHzgSjmHpjCMUWAsvnTwr5kiBwu8+JnoAL+0ZjuuP7RmVr\nZ12xtaO0wJanpRlbCB0+Jybj5l1GQMEyKHYZcQuhABcEDmcBMRpNo69IEOxWC/72nWvxxlQSu0ei\nJQFloFB1HMuUxhB6W1xlK3CZy6iWqWkZMY94NqezEIJV0jfX9QVUAWDsG42BEGBNb0B3POCyI5oW\nS2YhaOnwOTCVEORuqCYFoVCdXGQheLmFwOCCwOE0wHd/dwxDw0YTYmuHUoqxaAZ9Bv7+K1Z14fJV\n8jyQ4oAyIIuGx2EtsRDGylQpM/wuGyQKtW21GdjkMe3CqnXbGA2ZWdcXwNGJhNrrCJAthMF2r5qF\nxAi47ZiIZ5GTaFl3UIfPiZlkFpF05XnKWtrLWAhMEIpjH4sRLggcTp2khBy++sRB/PD3w035vOmk\nACEnlVgIjC++ax0uX9WJS1cYD4oyanBXrm0Fg3U8rSWwXFylDOiLx4pdRgCwtq8FEgUOhgpuo32j\nMaztC5Sc63fZkJdki6VcULnD54BE5eE/5oPKTuWn/vxWNYbABYELAodTJ0fGE6BULhRrBqwGoVyK\n6GCHFw/86daSNE1GcQtsSaIYj5WvUgbqm5pW6D9UWKxtVotaZVzOZQRAjSNEUgJGwmn1uO57aJrw\nlWtaV06MKsGEoM2r//uxWMhiH58JcEHgcOqG1QacnEmpTdYagdUglLMQqhFw2RHPFp5jKim7XcoJ\nDFDfXOXihnQMFkcwcr0MtLrR4rargrBf+bm+r9T9pd3xl7cQtPELc4Lw5nM7cPXabqzs1mdorekN\n4Oq13di0tNXU55zNcEHgcOqEDWkBmmMlMAuhbkFw23UWQqEGofzn1TMk59hkElYLKRGEVmVhLi5M\nAwBCCNb1BbBfCSwzYTC2EAo79XKLfae/tAaiGkvaPLjvI5tLLAGf04b7PrK57r/72QQXBM6C5iP3\nv4L7X3xjvh/DkIOhGAbbPQCaIwhj0TRcdou6sNZKwGXTxRBUQajoMqothkApxRN7x/Dmc9tLFv6g\nwbQ0Lev6AjgYiiOXl7BvNIqegEvn+lG/h2aBLxdU1lZJmxUETnW4IHAWNDuHZ/D6qch8P4Yhh0Jx\nbF3WhiVtbuxtioWQQV+Lu+4hLawFNoPNUu5uMY45AJo+Q4qF8MTeMfz4peGy5x8MxTE8ncK163tL\n3mO7+XLZOuv6WpDNSTg2mcS+0ZihdQCYiyG0uO2wKbUYbKYzp3F4FIWzYMlLFEkhj0i6cf98s5mM\nZzGdFLC6J4BkNo/dpxsXLaMahFoIKE3hKKUghCAUzcBmIejwlhcENkYzlhERy4j47M/3IJ4R8eZz\nO3Bel6/k/Mf3jMFCgLev6y55j/n7jVxGQME9tPNEGMcmE7h2fY/x93AXgtNOm/FnWSwE7T4HxmNZ\nbiE0EW4hcBYsbNcaXYCCwNInV/f4sb6/Badm0ogoM4DrZTSSrhgArkbALadrspqCUCyD7oBLnTVg\nhDbt9P4X30A0LcJuteCbvzlieP5je0PYtrzdMNOpmoWwvNMHl92Cn786AonKqajG36N0VoER7V4n\nCCm4vTiNwwWBs2BhGTPROhfa/9hxEjf+y4umq3Appdh5ImzqfJZhtKrHrzaa23s6VumSEn70h2G8\n45+eRyQlQMxLmIhnG7YQgEI/o2o1CABgtRB4HFaMhNP4wQtv4B3ruvFnb1mG/9o9qn5HxpHxOI5O\nJHDthlJ3EVBYwMulb1otBKt7Ath5IgzAOKAMAD6HDYRUjw10+J0IuOwVBY9TG1wQOAuWeIMWwm8P\nTmDXSBSTRRPHyvH6qQje8+0/4HeHJ6ueezAUR6ffiXafE+v75YWt1sDyzhNhHBqP46/+43WEohlQ\nCvQFG7EQ9C2wQ1VqEBh+lw2/fO00EkIOf3X1Stx+6XJ4HTZ845nDuvN+vWcMhADvMHAXARoLoUJP\nICYCLW47BlqNxc9iIfA7bVXTSVd2+bCsw1vxHE5tcEHgLFhY5ks0LUKSzPfaYagzBCYSps5nWTmv\nKjvYShwMxbC6xw9Azq5Z0ubGnhrjCOOxDJw2C549NIm7H9kHoDAHoR4K/Yzk2cqhqOwyqobPaUNO\norh+Qy9W9wTQ6nXgz96yDI/vDen6Dz2+J4Qtg23o8ht/5pbBNrxtdVdJnr+WdYqbaG1voGLwvNXr\nKGlTXcxnr12NB2/fVvEcTm1wQeAsWFj1rERR8yD4lJDDiZkUAOCYSUEIK8Vlr49U3unnJYoj4wms\n6varx87vD9ZsIUzGs7hqTTduuKAPvz04AaD+GgRAPxMhns0hJeRNxSR8LjssBPj0VSvVYx97yzIE\nXDZs//ke/PSVk3hm/zgOjcdxXZlAMHv2+2/bolY/G8EshHLuIsbX3nO+7nmMsFtLx2tyGoMLQgMc\nGIshI5pvCsYBXj0ZNr3b1xZLFTdtq8Zhpa0EIBdSmSGsxCp2j0QqxhGGp5PI5iSs1nTprCewPB7L\noCvgxD/cvEHN6GmKyygjqtZOtwlBeOeGXvzllSt1WUUtbju+8K51OBVO4XP/uQcf//EQAOAag3TT\nWljTG8A71nXjXRf0VTzv4uXthllOnNmFh+frJJHN4YZ/eRFfumE9PnjxOaau+fXuMditBG9fV36X\ndTZzYCyGm7/1B/yvm9bjw9uWVj1fKwiRlIglbebvdUjJAmr3Oky7jMJKF89ISsTJmRSWthv7pw+O\nya4o5jICChPM9pyOlm0+pyWRzSEp5NEdcMHrtOGBP92C105GGuqnwyp8Y+mcKghmLIRPXLbc8Ph7\nNw3gPRf1Y3g6hddPhWG3WqoGqavhsFnw3Q9vbugzOLMHtxDqJJYWIeapuqs0w9eePIjvPn98Fp/K\nmNdOhnHJV3+LTz/4mtq6eD5ghVL3PX9c7WZZCa0g1BpYPjAWh8dhxaUrOswLQkoEc2vvquA2OhSK\nwUKg28HWGlieYEVjATl9c6DVU3XXXA3tkBz2tzYTVK4EIQTLOrx498YBvPP8xp6Ps/DhglAnKUFe\nrLI5ydT5sYyIE9OphnPVa+WJvSF84L6XIeYlPLp7DFd//Xf4r12jNQ1EaRbsu5+cSeGJvaGq5yc0\njdoi6dr+bodCcazs9mNFtx+hWMZUa4ZISsDKLj+cNgt2VaiOPhiKY1mHV+e/DnocGGz34I/Hzc1G\nGI/JmU/lArT14LBZ4LZbEc8WLISuQPmiNA6nGO4yqhNW/JPNmYshHFDb/s5ekdVMUsC1/+95tHmd\nuHBJEH6XDfe9cBwXDATx/Y9uxlQii//58G586qevIZoW8SETbptmEk7K373L78R3nz+G6zb0VMw0\nKXYZmYVSioOhGN6xrgfndsq7+GMTCVywJFjxupmUgE6/E16nFbtHKgtC8ZB7QPav3/fC8ZJ5w0ZM\nxPUWQrOQW2CLSGRzaPc6ylb6cjhGcAuhTpJZRRBEcxaC2gc+Lc7a7vzkTArjsSwkieLR3aP43vPH\n8Y61PXjw9m3o8DmxuieA/7zjzegJuPDqyeqplc0mkpZdMp+6cgV2j0TxcpXddCKTQ6dfXjBrcRlN\nxrMIp0Ss7vGrbh0zbqOIMtT9giVyxlAuX/rv9tWTYZycSRmOsbxpYx/yEsVje8aq3muCWQgNunSK\nYUNyzKaccjhauCDUScFlZM5C2Kvkc+clWnMKZTEHQzFc9rVnMV1UcJVSPvdLN67Dri+8HX/8myvx\n7Q9dpHNt2JTAIFuQGiEl5HB0Il79RIVISkDAZcf7Ng2gw+fAd58/VvH8WEbe5brslpoE4aBaRRzA\n0nYPbBaCY5PVBSGcEtDmdeDCJUFkRAmHx/XXZMQ8/vpnu9AfdOPDbyq1rlb3BLCq249fvna66r0m\n4hm47BZ1HkGzYC2wQ8osZQ6nFrgg1ElSqM1CYANBACCSbMxttOtUBCdnUhieTumOMz+512GDxULQ\nHTAert7ld6oui0b44e+H8a5//r3hTtqIcEpEq8cOl92K2948iOcOTepGKhaTyIoIuOxocdtrGkDD\nCtJW9/hht1qwtN1T1ULISxTRtIigx4HzB2TXUrHb6Ju/OYJjk0n8w80bSuYAM27c2IdXT0Zwsujf\nTTHjsWzZfz+NwFpgh2IZUymnHI4WLgh1wnbjZoLKGTGPIxMJNU2xODPp+y8cx5/+8BXT955SZtom\niywNFteoNhu2K+DERLxxC2F4Kom0mDc9XCWSEtCi9Lt576YlAICXj02XPT+eycHnsiHodtQUVD4Q\niqHL71SHqZ/X5atqIUTTIiiVh7wMtnsQcNmwSyMIe0ai+O7zx3HL5gFctrJ8WukNSqbQI7tkK0HI\nSbj9x0P4Hw/t0p03Hsugy9/8gG/AbcdUPIuZpNBwhhFn8cEFoU5qCSofHo8jL1Fccl4HgFJBeOWN\nGTx7aFKdmFUNNsKwWBBUC6GKG6LL70IkJZp2d5VjXBEVs+2pI4qFAMjBVIfNgtFoeUslkc3B77Kh\nxW2vKah8KBTXFY2d1+XDiekUxAqWDPt30uZ1gBCCC5YEseuU7OYbj2XwmZ/tQofPgc9fv7bivQda\nPdgy2Ipfvj6KvERx10Ov46n943j20ITuvMl4tunxA0COIYyqk9K4IHBqgwtCnbAYQsaEy4gFlN+i\nCEKxP5wt8C8emTJ172nFQihOpWTPVF0Q5J3pZINWAsulN+vfD6cEtSMmIQT9QTdOVxDBeEYRBI/d\n9D1yeUlnjQHAuZ0+5CSKE9PlK5ZZURqb+nXBQBCHxuP415eGcdXXf4fh6SS+9t4LTPXev/HCfhyd\nSODjP9qBR3ePYW1vADNJQRfzGY9l0N3ElFOGthU0txA4tcIFoU6SNVgI+0aj8Dtt2DAgZ6aEi4rD\nppXXLxw1KQjJchaC/CyeKv1dWG56o26j8RoFIZISdQtqX9BV1iqilCKeEeFz2hF0mxeE4ekkhJyk\n6zNUyDSqIAiKBcIsmAuWBJGXKP7uV/uwpjeAJz59Gd5awVWk5foNvbBZCJ49NIn/9tbl+Oy1qwEA\nR5Q4BqtSno0aAe34SR5U5tSKKUEghFxDCDlECDlKCNlu8P5qQshLhJAsIeQzZq4lhHyREHKaEPK6\n8s91jX+duaOWGMK+0RjW9gXU+bDhIvcH2/H//uiUqT4/U3ElhiDoxSiVzcHjsFbtD8+KoRrJNMrm\n8ur3MFNsJ+YlJLI53dCTvhZ3WUHI5iSIeaq6jMwKwkHNnAKGWotQIY7AXEbs+S5e3oZLV3Tgyzet\nx4Of2FZTm+VWrwN3XH4u7rj8XGy/ZrXa/ZMJQnGVcjPRjp/kQWVOrVTNeSOEWAHcC+BqACMAdhBC\nHqGU7tecNgPgTgA31XjtP1FK72n8a8w9ZrOM8hLFwbE4PrD1HNisFgRcNt3ilhHzSGRzOK/Lh6MT\nCewfixnmuGthFkKxyygp5Ez1wim4jOrPNNKKiZnGcywG0OrVWghuTMSzEHISHDb93oQFqv0uGyiV\np4Blc/mqhVaHQnFYLUTXVsLrtKG3xVUx04iJGgtEB1x2/OvHLq76vcrxP96+Sv29J+CCz2nD0XFZ\nrFiV8my4jNj4Sa/D2vSUVs7ZjxkLYSuAo5TS45RSAcCDAG7UnkApnaCU7gBQvDJUvfZMJa0IQqaK\ny+iNqQTSYl5t9xv0OHRBZRY/uFHJTnmhShwhL1G1H1GxyyiZzcNXJcMIANp9TlhIYy4j5i4CzLmM\n2IIb1FgI/a1uUFqYQ6CFiR2zEMzch1KKp/ePY31/S0lb5GqZRjNJEXYrgbfCcJd6IUQWKNVCiM9e\nWwlmIXS3ND+llXP2Y0YQ+gGc0rweUY6Zodq1nyKE7CaE3E8IaTX5mQuCJCtMq2IhsIDyOqX5WavH\nrnMZMXfRmt4AVvf48eLRytO6IikBzKtUYiFkzVkIVgtBh89Z0WX07KEJPLG3fMXtuOZaMxlALBMp\nqPFx9yu9/40Cy2wWgs9pV1NVq1ki+0ZjOBiK472bBkreO7fTh2MTibIuuYgS8J6tRXSFVhBmqUoZ\nKMQQeECZUw/zGVT+NoDlAC4EMAbgH41OIoTcTggZIoQMTU5WH204V6Sy5oLK+0ZjcNgsqh876HHo\nZgQz90+7z4G3nNeBHW+EVevDCFaDABhYCEKubMFUMXItQnmX0T//5gj+92MHy77PLAS33WrKQmCB\ndF0MQREEozhCQuMyYiJSTXj+Y8cpOG0WtRZAy6alrUgKeTx3eMLgSn0G1GywotuHyXgWkZQwa1XK\nQKEFNk855dSDGUE4DWCJ5vWAcswMZa+llI5TSvOUUgnAfZDdSyVQSr9HKd1MKd3c2Wkuy2MuSJrs\ndrpvNKpWzALy3FmthcAW+A6fE5eu7ISQl/DKcPkeP9rURdZPSfvaY8JlBMiB5fEKFkIomsHJmZS6\nUy9mPJ6Bw2rBkja3SZeRYiF4SrNgjCyEmCIIPqc5l1FGzONXr5/GNet7DFNDr1nfg74WF77zO+P2\n4+GkWHWGbyOs6JKD3EcnErNWpQxwC4HTGGYEYQeAFYSQZYQQB4BbATxi8vPLXksI0Y5eejeAveYf\ne/5hhWkZMV+xWd3BsTjW9BSKpFqLYgjMZdTuc2DrYBscVgtePFLeEppSdtrdAacqSoykkKtag8CQ\n21cYC0JeomrRGWsDUcxELIuugBNBj8NUYRqrNNYuui67FR0+p7GFoFg/AZddvaaShfDkvhBimRxu\n2bzE8H271YKPXbocr7wxY9jYb7YtBBbkPjKRmLUaBED+7+sd67px+aquWfl8ztlNVUGglOYA/AWA\nJwEcAPAQpXQfIeSThJBPAgAhpIcQMgLgLgB/SwgZIYQEyl2rfPTXCCF7CCG7AVwB4K+a/u1mEVYE\nJlEgV8YvPZXIYjop6FIggx474pmc2v9nKpGFx2GFx2GD22HF5sHWioFlZiEsbfcaBJVz8JmcuNXl\nd2I6mTXsQzSVyKoDbA6MGfcaCkUz6Am40OK2m8oyCqdE2CykxKXVH3RVjCFog8qVhOdnQyMYaHXj\nTcvby55z65YlaHHb8T0DKyGcEtUMo9mgP+iG227FkfEEJuNZdM7SnAKrheC7H96MrctqGC/H4SiY\nWj0opY8BeKzo2Hc0v4cgu4NMXasc/3BNT7rASGncNdmcpLqEtBxWdtcrNUVSbBcaTYto9zkxncii\n3VdYiDYvbcU/P3vUMBUTkC0Kq4VgIOjG6bB+IU3V4DLqDLhAqVwUV9wmeUyT9bN/zNhCGI9nsKYn\nALfDir0ms4yCBkHbvqAbh8dL78HSTn0uGyyEgJDyLqNTMym8eHQKf3XVyoo1GF6nDR/ethT3PncU\nxycTWK7EdSilSlB59lxGFgvLNIpjPJbhO3jOgoRXKtcBpRRJIafuXLOicRCYLXQrewo58cz9weII\n00kB7d7CbnGg1VM2FROQd+9tXgd8Lpsuy4g9k+mgslKLYJRpFIqm1WctZyGoLiOTfYbCSdFwwe0L\nujEayZS43RLZHFx2C+xWC6wWAr/TpgvGa3l45wgIAd6zqXry20ffPAi71YL7XnhDPRbP5pCT6Ky6\njAA502j3SFSZpcwnmXEWHlwQ6iCbkyBRuREaAGTKBJYPjSfQ6rGjUzM9i+Xhs7z8qYSgm67V3ypn\n3oxEjNsnTyUEtHsd8DptSGZz6kKaFvOQKEwPaVcFwSDTiFkIl6/sxMFQrGT+cSKbQyKbQ7fiMkqL\neQhVguuRtGAYtO0PupEW8yXV26xtBSPocZS1EIZOzGBDfwsGWj0VnwEAOv1OvHfTAH7+6ojaUpu1\nI59NlxEAnNftU78DH23JWYhwQagD5rtnC1wlC2FFt1/nJmktCpDK4xY1xVosNz9s3NJhOimPZ/Q5\nbchJVM1yYhlHZgrTgEIOvFFgORTNwGGz4M3ndiAjShguagqnbb3Q4jFXNCZPIytdcMulnsYzOTWF\nEpD/1uViCKORDJa0VRcDxjs39ELISWp760LbitlzGQHAyq6C63C2gsocTiMsOkHIiHm8MVW+yZkZ\nWIZRm7LAGaWeUkpxeDyua7IGAEG3fE04JUBSqo61MYTeYPlUTECOIbT7HGpFLRMn9tOshcCsFiOX\n0ZgybWutUl1d7DYKMUHwuzQpoZX7GYXL+OjLFaexWQiMci2wKaU4HUmrn2OGdUprkD2n5fbWMwZV\n1LPBiu6C65BbCJyFyKIThH965jCu/+YLFXvjV0MVBG95QQjFMohncmpjM0bQW7AQomkReYnqYghO\nmxVdfmdZC4ENcGfppcwySJpsfc1w2Cxo9dgNXUYsg+i8Lh+sFlIiCExEultchjUCM0kB3/zNEV0G\nU3kLQRbAYguBzUJglMtmmk4KEHIS+mooxGpx27G03YO9iiBENLMQZpOBVg+cSqLAbFQpcziNsqgE\ngVKKx/eEkBLyZYO2ZmCLrxpDMHAZHTLIMAIAv9MGq4UgnBJ0Vcpa+luN5wSkhBxSQh7tPocaPE6o\nFoK5aWlauvwuQ5fRWCyN3hYXXHYrzu304kBRptG46jIyFoQn9obw9acPqy6ZtJBHNicZxhDalJnJ\nxQIYz4jwa2IILW5jlxETkr4aLAQAWN/fgt0jsiCEWQxhll1GVgvBuZ0+uO288RxnYbKoBOHIRAIn\nZ+Rg7UiZHbgZWMppawUL4YgyoL1YEAghcmZOWlSrlLVBZwBlB8ewIrYOr8ZCUMSpVgsBMB6lKUkU\n49EselrkBXZNb6DEQhiPZeF1WOFz2jRB8sJizRZp1oq6uLW0FkKInGkULbIQilxGQWVITnE2EhMS\nFow3y/n9LTgdSSOcFBBOCbAQfevo2eLCc4I4t8vLG89xFiSLShCe3j+u/j4SrjwEvRKqhcBiCEYW\nwngcnZq5vlqCHjsiKUHtdNpeLAitboxFMiWN2NggnXafQ134E0UxBK/JGAIgZ9xMxvSW0kxKgJCX\n1LYSa3sDGItmdEN9xuMZtXbByEJgi/uhEkEwXnBlAdQ/B5uWxgi6HchLtKShHxPOWmIIALBBE0cI\nKzUS1eZINIO/u34tfvKxbbN+Hw6nHhaVIDy1fxxrewMgpDELgTWfq2QhGAWUGa0eB8JJUde2QstA\n0A0hL2Eyod+9sypllmUEFIQgVYfLqDsgu4y0whMqmse7prc0sDwezahBUZYJpBMEZiEoriaW3tni\nNvbRFw/KkSSKhJDTuVVayjS4G41k4HFYTY221LJOJwiz28dIi9thVTOzOJyFxqIRhIlYBrtORXD9\n+b3o9rsaEoRCDEH+H7s4hiBJFEfGEyXuIgbr/zOdyMJCSl0pai1C0TMWLAqHuvAzQUjUYSF0+Z3I\nSVTXW4nVIPQWCcJ+rSDEM2rzNJtV7tqpXajZZxwMxUApLYyn9JaxEFrdmIxn1b9jUsiBUsCvceGU\nS28djaTRF3TX7IJhgeU9I1G19TWHs9hZNILwzAG57fFVa7ox0OpuyGWkxhDKpJ2OhNNIi/mSDCOG\n6jJKCmjzOmAtclX0B+Wc+uI4Aos5tHudmqCy/CypemII/tJaBFalzCyETr8THT6nGlimlKrdOhkB\nTQaQJFGMRTIIuGyIZXIIxTIVYwhAISDMrBNt2wpGuY6no9HaUk61bOhvwZ7TUcwkRS4IHA4WkSA8\nva+4ODgAABSySURBVD+Ec9o8WNntUwShcQuhnCAcUltWlHMZ2RFOCZiKZ3UppwxmIRRn3kwnBHgd\nVrgdVk3aKbMQ8nBYLYb9j8rB3D5aQRiLZmCzEHRonmtNrx+vngwjl5cQTYsQcpIubVI783g6Kccg\nLlMG0h8ci6vvlXPrFKeeaqelMYJlLITT4XTNGUaMDUpg+dRMatYzjDicM4FFIQjJbA6/PzaNq9Z0\ngxCCgVYPQrGMYadPM6SFPFx2C9xKcVjxkBzWw2hFVzkLwYGMKOF0JF0SPwAKMwBOF7WvmE5m0aG0\nnLAri78aQxByphvbMQr9jAoB3VBUDhhrA6zv37IEb0wlcf/v3yjMAw5o23EUUkLZon6F0rztYCiO\ncFKA224tGWvJKC5OK0xLqxxDyIh5TCcF9Afry+lngeVENjfrbSs4nDOBRSEILxyZhJCTcPXabgDA\nQKsbeYmqFbe1khRy8DpsapFRpmiM5uHxOPqDbp0PXAvb7R6fTJZkGDH6DbqZTiWyaNcsXD5nocFd\nIpurKX4AGLuMWJWylus39OLta7vxj08dxsvHpwFA5zLSWghjistpVY8ffS0uHArF5NbSFXbgfUE3\nnDaL6paKq9PSNL2MlIB0RFMRXW8NAoMFloHy7iwOZzGxKATh6f0TaHHbsWVQHttcLmhrFtZmmhAC\nh81SYiEcCsXLxg+AwuKTFvO6PkZajIrT5LYVBQHxOq26LKNaMowAOePF77RhUhtDiGVKxi8SQvDl\nm9bDZbfifz92AIB+IpdWEFj6aF/QjVU9fhwMxdXW1+WwWy3YeE4QO5RJcUwQtL2MXHbZItJnM8n3\nqjeG0OK2Y7BdjtdwlxGHs0gE4W+vX4Pvf3QzbMrMAtYVs15BYBYCADhtFmSLLITRSBrnVGi2pk1x\n7KhiIWgLseTOqIWF1euwISkUMnNqCSgzVnT78PLxaVBKQSnFWDRdYiEAcquFL7xzrRov6fQXnrvF\nY0c0JReNjUXScNnlthirewM4NpnAVCJbNa1z67J27BuNIp4RDYPKhBBZeDQuI+ZSq9dCAOSKZWD2\n+xhxOGcCi0IQWr0ObBksTJBiQcx6M41SQl6NH7jsVl1QWZ5LkNctZiXPo1l82sv4rgda3UgKeXVH\nLDfC0wehWQtsQI6T1OoyAoCbLxrAwVAce05HEU2LyIiSWqVcem4/rljViT6lrQWjxW2HkJeQESWM\nRtPoa5HTQFf3+CHmKfaPxaq6ZC5e1gaJAjtPhJHIsmlpehFp9zp0VtPpSAaENDZQnsURZruPEYdz\nJrAoBKEYp82K7oCzfpeRkC+yEPTT0/ISrdh1VLtbrhRDAApWTCQtQqLQWwg6QajdZQQA77qgD06b\nBQ8NnSqpQSiGEILvfHgTfvHnl+i/j7swBW40klF37Gx0qJinVS2EjecEYbMQvPLGDOKZHAgBPEVB\n6G3L27FjeEatVxiNpNHtdxlOqzPLVWu7sfGcYEUXH4ezWFiUggDIbqN6LYRkNgePYiE4bRadhcCC\nvJUml+kshAoxBKCQeWPU5sLntDYUVAbk3f2163vwq9dHMay0Ba+045bFVP++tkZA63Ja3uGD3Spn\nK1UTBI/Dhg0DLaog+Jy2klYSl6/qREaU1MD2aCRdcw+jYs7t9OEX//0S7jLicLCoBcG4gZwZUkJe\n9dc7bVZdUFntKVRBEFx2K1x2+U9f3NiOUTwoR1ulzPA6bGqX01SdMQQAuGXLEsQzOfzwD8MA5FYS\ntcAEYSqRxUQ8q1oIDpsF5ypzi81k8Wxd1oZdIxFMJrKG3UC3LW+H02bBc4cmARSqlDkcTnNY1IIw\nFqleiyDmJdz77FG1EhiQF99CDKGchVDZfcPcLOUsBLUttCJaLBOow1cuhpCvuQ6BsW1ZO5a0ufHK\nGzOwWoguYGwGtvs/FIqD0kKMBii4jczswC9e1gYxT/GHo1OGKbsuuxVvPrcdzx2agCRRxT3F5wpw\nOM1i0QpCf9CDnEQxbjAPQMuuUxH83ycP4bcHJ9RjyWxenVjmtFl1vYwKcwkq79aDHjvcdmvZWAMh\nRM00Goum8bUnDqHD58ASzdxgn9OGpJCDkJMg5CX46nAZAYDFQvC+TUsAyMVqxa00qsEsBNYAT7tr\nZ4JgJq1z09I2EAKEU6KuSlnLFau7MDydwtCJMIS8VHfKKYfDKWXRCsIAq0WYqRxHiBZV4EoSRVrM\nqwu5s8hCMOMyAmQXSjnrgNHf6sHhiTg+8oNXEE2LeOBPt6qWCbuHRKEO2vE0MHTlPZsG6s7YCTBB\nCMmC0KtxOW1cItd+DLRWn3nc4rZjTY/cTK9cltblK+UK6J/88QSA+msQOBxOKYt2bNOApjjt4grn\nxTJMEOQMnLSobzNdXIdgJqgMAG9b3WU4vlJLf9CN5w9PwmGz4Ed/ulXNmWcwtxRrJ1HNTVXtXh/e\ntrQkYGwGv9MGCwEOK0OBtG6cN53bjhf+5xVYUqEuQ8vWZW3YPxYrW+V9TrsHyzu9eHxPSLkXFwQO\np1ksWkHoC5qrVo6l5QWe+fJZYzvVQqgjqAwAn7hsedVnPK/LBwsB/vkDG/Gmc9tL3mf3YL2I6g0q\nM/7+xvV1XWexEATcdmVusr3EDWZWDAA5jvDAH4YrCuoVq7rwgxffAMAFgcNpJovWZeSyK8PsI7W5\njFjra0+1oHKd/nwtH962FL/76yvwjnU9hu8zAWBxkHrSTpsFiyPUmqFUzJZlcgFhwF3+u1y+Su6k\n6nPadO0tOBxOYyzq/5vMtMGOqT16ylsIxkHl+t03DIfNUnF3zXbRk02yEBoh6LbjBNBw1k+Hz4mv\nvfd8XWV5MVuXtcFtt6K/jsE4HA6nPItcEDx4/VSk4jkshhBJiUhmc+r4TF0MQRtUFnJw2ixq36TZ\nRLUQlBiCx9G4CNULCyw3w4Vzy+YlFd932qz4yJuWqt1mORxOc1jUgtDf6sZje8aQl2jZVEsWQwDk\n1s6smVy5LKNENlc1oNwsWBCZBafn6r5GMJdRb4MuI7N87ro1c3IfDmcxsai3WH1BN3IS1bV/LiaW\nEdX2C6cjGaSyzGWkxBBsVuQlqha4pbL1VwzXSomF0AQ3Vb2w4jReKMbhnLksakFgPf3HKwzKiWVE\ntf3CaKRgIXg1FgIAZBQrIZHNz7kgsAE3C8FC4Fk/HM6Zy6IWBDYGsqIgpHNY0e2H1UJwOpxWW1h4\nnIVKZQBqx9NkNtdQPUAtMFGaTmZBCOAuM6JyLmCtOLggcDhnLos6hsCKsCq1r4imRbR57OgJuDAa\nSasVtNr21wDUOEJSyM1Zb32rhcBttyIt5uFz2uY14+aGC/vgsFnQ18BsAg6HM78saguhw+eEheiH\nzGuRJIp4RkTAbUdf0IXTkTRSWblXP+tWygbFZFWX0dzFEICC22g+M4wAWVw/+uZBngbK4ZzBLGpB\nYJ09y7mMkkIOEgUCLjv6gm6MKllGHrtVXfiYhZDRuozmsECMuafmM37A4XDODha1IADyzpZl6RQT\nU2b7trjt6A+6EYpmEM+IuiZyLKisuozmMKgMaCyEecww4nA4ZweLXhC6/K6yFgKrUg64begLuiHm\nKU5Mp9TW14A+qCzPU567oDJQEIT5bFvB4XDODha9IHQHyruMVEFw2dU2y0cnErrmbS6NhZAS8qB0\nbltIMFcRdxlxOJxGMSUIhJBrCCGHCCFHCSHbDd5fTQh5iRCSJYR8xsy1hJA2QsjThJAjys/Wxr9O\n7fQEXAinRF3HUkZUtRDsajrldFLQ9SliFkJGzJvudNpMWDC5kVkIHA6HA5gQBEKIFcC9AK4FsBbA\nBwgha4tOmwFwJ4B7arh2O4DfUEpXAPiN8nrOYamnEwZxBBZDkIPKhXRKt8ZC0Kadmp2F0EwKFgKP\nIXA4nMYwYyFsBXCUUnqcUioAeBDAjdoTKKUTlNIdAMQarr0RwI+U338E4KY6v0NDdCnFaUbDarQx\nBL/LrrZaNowh5CTT4zObSSHtlFsIHA6nMcwIQj+AU5rXI8oxM1S6tptSOqb8HgLQbfQBhJDbCSFD\nhJChyclJk7c1D7MQQlEjC0EWBDa9i7mNjGMIedVCaEbra7OoQWXuMuJwOA2yIILKlFIKgJZ573uU\n0s2U0s2dnZ1Nv3d3hX5GsXQOfqdN7YTKAsvGMQSpEEOYhzoE7zwXpnE4nDMfM4JwGoC2Qf2AcswM\nla4dJ4T0AoDyc8LkZzaVVo8dDqsF40YuI6VKmdGvzGHWDrp3aiwENjxnPlxG3ELgcDiNYkYQdgBY\nQQhZRghxALgVwCMmP7/StY8A+Kjy+0cB/Mr8YzcPQgi6Ak7DoHI0LcKvGdHIXEZaC8ChDMLJivMb\nVJ5LNxWHwzk7qbpyUUpzhJC/APAkACuA+yml+wghn1Te/w4hpAfAEIAAAIkQ8mkAaymlMaNrlY/+\nKoCHCCEfA3ACwC3N/nJmkauVjYPKWguhEEMoLL4WC4FDmZqWnI8YgoMXpnE4nOZgahWhlD4G4LGi\nY9/R/B6C7A4yda1yfBrAlbU87GzRHXDiUChecjyWyalxAwDoV1JPi90zTpsFGTGPRFa2FuZyce5R\nuot2BXiXUQ6H0xgLIqg838jtKwyyjNIiAu7C4r6y24+1vQGs72vRnee0WVULweOwwlJmHOdssL6/\nBc/cdRkuXBKcs3tyOJyzE+5ngLzLTmRzJfOQYxkRAVfBZeR32fHYX15acr3TZkE2J/cymo/g7nld\n/jm/J4fDOfvgFgIKk9O0cxEkiSKRzeliCOVw2i1qpTLvKcThcM5UuCAA6PazWoSC2yieyYHSwqzg\nSrhsVmSVXkY824fD4Zyp8O0sCgFZbfsKVqUccFX/EzELIStKPNuHw+GcsXALAQWXkTb1VNvptBpO\nm0WtQ+AuIw6Hc6bCBQFysNjrsOr6GRUsBDOCYEU2l0dKmNt5yhwOh9NMuCAodAdcuvYVsbTS+tpd\nfYF32S3IiBISczw+k8PhcJoJFwQFuX2FUQzBvIWQzM7t+EwOh8NpJlwQFOT2FRqXUY0xhJSQR1rk\nFgKHwzlz4YKgwPoZyZ24ZUEgBPCbWOCddgvCKQEAn23M4XDOXLggKHQHXMjmJDV2EMvIsxDMtKFw\n2awQ87KQcAuBw+GcqXBBUBj8/+3dW4hVVRzH8e/PGUcdBe+KOZKTyJQFZkRqN7SCvEQGvRgIPdRb\nkEUQib30HlEPFYRdpCIfTEp8iG5K0INlF8LUKcuyvEt4wS6a/XvYa3BzcqbjeI6HvffvA8Ocvc4e\nzvrNZf1nr7XP3uM7Adhx4ATw3yudDqTvngjggmBmxeWCkMy9YjxD28SW77L79NRex2ggfXdNA9/s\n3syKywUhGTWsnRu6x7FlV3bf5hN//F3XKaeQLSr38TuVzayoXBByFvZMovfQSfYf++OCjhCGDz13\nVOApIzMrKheEnAU9EwHY0nuE4xeyhpA7QvBZRmZWVC4IOTMmjqJr7Ag29x7OFpXrXUPworKZlYAL\nQo4kFvRM5NPdRzl1+mxdl76G2kVlFwQzKyYXhBoLeybx++mzQH3XMYLsWkYAQ3TusZlZ0Xj0qjF/\nxng60prAhZ52OrKjHenS3U/ZzKyRXBBqdHa0M7d7HFDfdYzg3KKy1w/MrMhcEM5jYc8koL67pUHu\nCMFvSjOzAvO/tOdx73VdHDrxJ7Onjalr/751Ay8om1mReQQ7j9GdQ1m15Kq69z93hOBvp5kVl6eM\nGqDvfQguCGZWZC4IDdC3qOwpIzMrMheEBui7lpEXlc2syFwQGqCjzVNGZlZ8HsEaYMgQ8eTSq7h5\n5oRWd8XMbNBcEBrkwVuuaHUXzMwuiqeMzMwMcEEwM7PEBcHMzAAXBDMzS1wQzMwMcEEwM7PEBcHM\nzAAXBDMzSxQRre5D3SQdAX4e5JdPAI42sDtFUcXcVcwM1cxdxcxw4bkvj4iJ/7dToQrCxZC0LSKu\nb3U/LrUq5q5iZqhm7ipmhubl9pSRmZkBLghmZpZUqSC81OoOtEgVc1cxM1QzdxUzQ5NyV2YNwczM\nBlalIwQzMxtAJQqCpEWSeiXtlvREq/vTDJKmSdosaYekbyWtTO3jJH0g6fv0eWyr+9poktokfSVp\nU9quQuYxktZL2iVpp6T5Zc8t6dH0u71d0luShpcxs6RXJB2WtD3X1m9OSavS2NYr6c6Lee3SFwRJ\nbcDzwGJgFnCfpFmt7VVT/A08FhGzgHnAQynnE8BHETET+Chtl81KYGduuwqZnwPei4grgdlk+Uub\nW9JU4GHg+oi4BmgDllPOzK8Bi2razpsz/Y0vB65OX/NCGvMGpfQFAbgB2B0RP0bEaWAdsKzFfWq4\niDgQEV+mxyfJBoipZFnXpt3WAve0pofNIakLWAqsyTWXPfNo4FbgZYCIOB0Rxyh5brI7PI6Q1A50\nAvspYeaI+AT4raa5v5zLgHUR8VdE7AF2k415g1KFgjAV+CW3/WtqKy1J04E5wFZgckQcSE8dBCa3\nqFvN8izwOPBPrq3smbuBI8CraapsjaSRlDh3ROwDngb2AgeA4xHxPiXOXKO/nA0d36pQECpF0ijg\nbeCRiDiRfy6yU8pKc1qZpLuAwxHxRX/7lC1z0g5cB7wYEXOAU9RMlZQtd5ozX0ZWDC8DRkpakd+n\nbJn708ycVSgI+4Bpue2u1FY6koaSFYM3I2JDaj4kaUp6fgpwuFX9a4KbgLsl/UQ2FXibpDcod2bI\n/gv8NSK2pu31ZAWizLnvAPZExJGIOANsAG6k3Jnz+svZ0PGtCgXhc2CmpG5JHWQLMBtb3KeGkySy\nOeWdEfFM7qmNwP3p8f3Au5e6b80SEasioisippP9XD+OiBWUODNARBwEfpHUk5puB3ZQ7tx7gXmS\nOtPv+u1k62RlzpzXX86NwHJJwyR1AzOBzwb9KhFR+g9gCfAd8AOwutX9aVLGm8kOI78Bvk4fS4Dx\nZGclfA98CIxrdV+blH8BsCk9Ln1m4FpgW/p5vwOMLXtu4ClgF7AdeB0YVsbMwFtk6yRnyI4GHxgo\nJ7A6jW29wOKLeW2/U9nMzIBqTBmZmVkdXBDMzAxwQTAzs8QFwczMABcEMzNLXBDMzAxwQTAzs8QF\nwczMAPgX1SBGnYPkipUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1f06ec32940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['training', 'validation'], loc = 'upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Validation results does not improve since it did not show signs of overfitting, yet.\n",
    "<br> Hence, the key takeaway message is that apply dropout when you see a signal of overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 9952/10000 [============================>.] - ETA: 0s"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy:  0.1135\n"
     ]
    }
   ],
   "source": [
    "print('Test accuracy: ', results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Model Ensemble\n",
    "- Model ensemble is a reliable and promising way to boost performance of the model\n",
    "- Usually create 8 to 10 independent networks and merge their results\n",
    "- Here, we resort to scikit-learn API, **VotingClassifier**\n",
    "- Doc: http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcRs1CBSEtpp5yj6SJ5K_nHd1FNfyEYa9KLjWfoMY_v7ARTq3tdpVw\" style=\"width: 300px\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "from keras.wrappers.scikit_learn import KerasClassifier\n",
    "from sklearn.ensemble import VotingClassifier\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_train = np.argmax(y_train, axis = 1)\n",
    "y_test = np.argmax(y_test, axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def mlp_model():\n",
    "    model = Sequential()\n",
    "    \n",
    "    model.add(Dense(50, input_shape = (784, )))\n",
    "    model.add(Activation('sigmoid'))    \n",
    "    model.add(Dense(50))\n",
    "    model.add(Activation('sigmoid'))    \n",
    "    model.add(Dense(50))\n",
    "    model.add(Activation('sigmoid'))    \n",
    "    model.add(Dense(50))\n",
    "    model.add(Activation('sigmoid'))    \n",
    "    model.add(Dense(10))\n",
    "    model.add(Activation('softmax'))\n",
    "    \n",
    "    sgd = optimizers.SGD(lr = 0.001)\n",
    "    model.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = ['accuracy'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model1 = KerasClassifier(build_fn = mlp_model, epochs = 100, verbose = 0)\n",
    "model2 = KerasClassifier(build_fn = mlp_model, epochs = 100, verbose = 0)\n",
    "model3 = KerasClassifier(build_fn = mlp_model, epochs = 100, verbose = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ensemble_clf = VotingClassifier(estimators = [('model1', model1), ('model2', model2), ('model3', model3)], voting = 'soft')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ensemble_clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 9088/10000 [==========================>...] - ETA: 0s"
     ]
    }
   ],
   "source": [
    "y_pred = ensemble_clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.3045\n"
     ]
    }
   ],
   "source": [
    "print('Test accuracy:', accuracy_score(y_pred, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Slight boost in the test accuracy from the outset **(0.2144 => 0.3045)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "|Model           | Naive Model | He normal  | Relu        | Adam        | Batchnorm  | Dropout   | Ensemble   |\n",
    "|----------------|-------------|------------|-------------|-------------|------------|-----------|------------|\n",
    "|Test Accuracy   | 0.2144      | 0.4105     | 0.9208      | 0.9248      | 0.9154     | 0.1135    | 0.3045     |\n",
    "\n",
    "<br>\n",
    "It turns out that most methods improve the model training & test performance.\n",
    "Why don't try them out altogether?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
